Together AI releases continuously compiling inference optimization system TKCC

1. 16:02
   MarkTechPost

   Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

   Together AI introduced Together Compiler (TKCC), an agent-driven continual compilation loop that uses coding agents to rewrite a model's forward pass and re-optimize inference as production traffic changes, already running on several of its serverless endpoints. Tested on models including Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B across NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2, it reported up to 2.6x higher TPS than the best published InferenceX configurations in under 24 hours per setting, with gains ranging from about 4% (GLM 5.3 on MI355X, 463 to 480 TPS) to 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1) versus a 633 TPS baseline. The system starts the engine once and swaps forward function pointers over RPC, using a fast 12-layer tier and a slow 60-layer tier that tested 8.1x more candidates (1,354 vs 168) and found a solution 7.2% faster, with results self-reported.

1. 08:00
   Together AI Blog

   Polymorphic Inference: self-tuning inference for open models

   Together AI introduces "Polymorphic Inference," an approach for adapting inference to fast-changing model architectures, traffic shapes, and hardware, contrasted with the older "Static Inference" paradigm. The company announces Together Compiler, an AI-agent-driven "Continuous Compilation" optimization loop already running in production on some serverless endpoints, and previews upcoming work including self-serve compilation, shape-shifting speculative decoding, the ThunderAgent Router, post-training, and TogetherLink workload-aware model routing.

2. 08:00
   Together AI BlogFeatured

   Together Compiler: Continual Compilation for AI inference

   Together AI introduces Together Compiler (TKCC), a continual optimization loop that uses high-throughput agents to generate inference-engine optimizations, built on "persistent engines" that hot-reload candidate forward passes without restarts and a two-tier engine design balancing measurement throughput against fidelity. TKCC reports up to 2.6x higher TPS across frontier open-source models in under 24 hours, state-of-the-art low-latency results on SemiAnalysis' InferenceX benchmark (1111 TPS on MiniMax M3.1-Flash, 1205 TPS on DeepSeek v4.1-Flash), and 1.7x over open-source megakernels on Llama 1B at batch size 1, enabled by 252x higher experiment throughput than baseline agentic systems. The post also says TKCC generalizes to weather modeling, PDE, time-series, and biology workloads, and is serving trillions of tokens today.

3. 08:00
   Together AI BlogFeatured

   Together Compiler: Continuous Compilation for AI inference

   Together AI introduces Together Compiler (TKCC), a continual agent-driven system that recompiles inference engines as traffic drifts, using persistent engines, a two-tier fast/slow engine setup for throughput-fidelity balance, and a custom cost model for parallelism selection. TKCC reports 1.3-3x higher TPS across frontier open-source models in under 24 hours, state-of-the-art low-latency results of 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash on SemiAnalysis' InferenceX benchmark, and 1.7x over open-source megakernels on Llama 1B batch size 1, enabled by 252x higher experiment throughput than baseline agentic systems. It also generalizes to weather modeling, PDE, time-series, and biology workloads.

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.