Buoyant open-sources vLLM inference control plane Kinference
Understand this story
Buoyant has open-sourced Kinference, an inference control plane for vLLM and Kubernetes. The code is released under the Apache v2 license and is now available on GitHub. According to Buoyant, Kinference provides token rate limiting, tenant usage and cost tracking, KV cache-aware load balancing, and inference policies. These feature descriptions come from the publisher, and no independently verified performance data is available yet.
AI overview based on source reports · Updated 1 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- vLLM BlogFeaturedIntroducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes
Buoyant announced Kinference, an Apache v2 open source control plane for building inference platforms on vLLM and Kubernetes, offering token rate limiting, tenant usage and cost tracking, KV-cache-aware load balancing, and a YAML-based inference policy engine. The post reports benchmarks in which Kinference cut median time-to-first-token by 95% and median end-to-end latency by 45% versus standard Kubernetes load balancing on one workload, and served 7.4× the goodput on a shared-prefix workload and about 3.5× on multiturn. It also covers per-tenant inference metrics and Helm-based installation steps.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 5Nov 5 10:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.