Author publishes tutorial on choosing an inference stack for voice AI
Understand this story
Camilo Quintas publishes the first part of a tutorial, using a fictional bicycle rental voice agent as an example to discuss how to choose an inference stack when migrating voice AI from token-billed closed-source APIs to open-weight LLMs. The article outlines three options: shared or serverless APIs, managed dedicated deployments, and self-managed GPU servers, and mentions open-weight model families such as Kimi, GLM, and DeepSeek, as well as OpenAI's Fast mode (formerly Priority processing), which trades a per-token premium for faster processing. The author states that the token rates in the article are illustrative only, and that the Together prices used for the alternatives are public quotes verified on November 11, 2026.
AI overview based on source reports · Updated 21 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIFrom Closed to Open-Weight LLMs: Choosing the Inference Stack for Voice AI
Part 1 of a series uses a fictional bicycle-rental voice agent to examine migrating from a token-priced proprietary LLM API to open-weight models, covering the model/engine/hosting layers, shared versus managed dedicated versus self-managed deployment, and workload profiling. It walks through an illustrative workload (1,200 conversations/day, 20 requests each, 8,000 input and 200 output tokens per request), a 1,500 ms p99 TTS-handoff latency target, prefill/decode and prefix-caching considerations, open-loop load testing for goodput, and a cost comparison in which a two-replica B200:4 schedule at Together's $8.99/GPU-hour comes to $359.60/day versus $278.40/day for the existing API.
Activity for this story
- Current comparable activity
- 6
- Peak comparable activity
- 10Nov 20 00:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.