Last year we built megakernels. This year we deleted them: The road to Together Compiler
Last year we built megakernels. This year we deleted them: The road to Together Compiler
Together AI describes how, after its megakernel work reached only one model, one batch size and one customer in production, it found that default Torch Inductor kernels plus CUDA Graphs and Programmatic Dependent Launch (PDL) recovered 98% of megakernel performance on Llama 1B on H100 without writing a kernel, and that "multi-kernels" with agent loops beat the old megakernels by up to 1.7x at over 1100 TPS. The post also details three Inductor limitations—heuristic flag selection, misleading isolated benchmarking, and missed fusion opportunities—and argues that megakernels' fixed launch envelope makes them suboptimal on newer hardware and architectures. These findings shaped Together Compiler, the company's framework for continual AI inference optimization.
Useful for readers tracking inference-optimization tradeoffs, as it quantifies how much of megakernel performance simpler primitives recover and where compiler heuristics fall short.
Source: Together AI Blog · together.ai