iCreatAI News
Skip to content
Pragmatic Engineer· Gergely Orosz·· 9 hr agoAI score58

Inference engineering for software engineers

Inference engineering for software engineers

AI summary

The article explains how companies move from frontier lab APIs to open models via inference providers such as Modal, Fireworks AI and Baseten, covering when to switch (cost, latency control, fine-tuning) and how to choose a model, provider and deployment strategy. It describes a three-stage graduation path from multi-tenant API replacement to dedicated GPUs, and details challenges like TTFT tail latency, cold starts, KV cache and prompt caching. It also notes frontier lab API uptime figures (Anthropic 99.5%, OpenAI 99.97% API) and GPU scarcity driving long-term commitments.

Source: Pragmatic Engineer · newsletter.pragmaticengineer.com