iCreatAI News
Skip to content
Together AI Blog·· 2 days agoFeaturedAI score60

Last year we built megakernels. This year we deleted them: The road to Together Compiler

Last year we built megakernels. This year we deleted them: The road to Together Compiler

AI summary

Together AI describes how, after its megakernel work reached only one model, one batch size and one customer in production, it found that default Torch Inductor kernels plus CUDA Graphs and Programmatic Dependent Launch (PDL) recovered 98% of megakernel performance on Llama 1B on H100 without writing a kernel. The post details three Inductor limitations (heuristic flag selection, misleading isolated benchmarking, and missed fusion opportunities) and argues megakernels are rigid across hardware and modern architectures. It frames these findings as motivation for Together Compiler, its framework for continual AI inference optimization.

Why it matters

Useful for readers tracking inference-compiler design, offering concrete ablations of megakernel gains and a critique of Torch Inductor's heuristics and benchmarking.

Source: Together AI Blog · together.ai