iCreatAI News
Skip to content
Hacker News AI· cqm89·· 21 hr agoAI score22

From Closed to Open-Weight LLMs: Choosing the Inference Stack for Voice AI

From Closed to Open-Weight LLMs: Choosing the Inference Stack for Voice AI

AI summary

Part 1 of a series uses a fictional bicycle-rental voice agent to examine migrating from a token-priced proprietary LLM API to open-weight models, covering the model/engine/hosting layers, shared versus managed dedicated versus self-managed deployment, and workload profiling. It walks through an illustrative workload (1,200 conversations/day, 20 requests each, 8,000 input and 200 output tokens per request), a 1,500 ms p99 TTS-handoff latency target, prefill/decode and prefix-caching considerations, open-loop load testing for goodput, and a cost comparison in which a two-replica B200:4 schedule at Together's $8.99/GPU-hour comes to $359.60/day versus $278.40/day for the existing API.

Source: Hacker News AI · github.com