Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

Together AI introduced Together Compiler (TKCC), an agent-driven continual compilation loop that uses coding agents to rewrite a model's forward pass and re-optimize inference as production traffic changes, already running on several of its serverless endpoints. Tested on models including Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B across NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2, it reported up to 2.6x higher TPS than the best published InferenceX configurations in under 24 hours per setting, with gains ranging from about 4% (GLM 5.3 on MI355X, 463 to 480 TPS) to 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1) versus a 633 TPS baseline. The system starts the engine once and swaps forward function pointers over RPC, using a fast 12-layer tier and a slow 60-layer tier that tested 8.1x more candidates (1,354 vs 168) and found a solution 7.2% faster, with results self-reported.