Buoyant open-sources Kinference inference control plane
Understand this story
Buoyant has open-sourced Kinference, an inference control plane for vLLM and Kubernetes, with the code released on GitHub under the Apache 2.0 license. According to Buoyant, it provides token rate limiting, tenant usage and cost tracking, KV cache-aware load balancing, and inference policies; these feature descriptions come from the publisher, and there is no independently verified performance data yet. According to MarkTechPost, the current version is v0.1.0-alpha.7, written in Rust, and configured via 5 Kubernetes CRDs. The report also notes that the project is still in the alpha stage, and rate limiting, classifiers, and Anthropic API support have not yet been released.
AI overview based on source reports · Updated 4 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- MarkTechPostBuoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Buoyant, the company behind Linkerd, has released Kinference, an Apache 2.0 inference control plane (v0.1.0-alpha.7, written in Rust) that sits in front of vLLM workers on Kubernetes to apply routing policies, KV-cache-aware load balancing and per-tenant metrics, configured through five CRDs. In Buoyant's self-reported benchmark using three L40S GPUs per pool, median TTFT on shared-prefix traffic fell from 6.88 s to 0.31 s versus plain Kubernetes Service routing, while prompts without shared prefixes showed no significant gain (6.92 s vs 6.83 s). The project is in alpha, with rate limiting, classifiers and Anthropic API support not yet shipped.
- vLLM BlogFeaturedIntroducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes
Buoyant announced Kinference, an Apache v2 open source control plane for building inference platforms on vLLM and Kubernetes, offering token rate limiting, tenant usage and cost tracking, KV-cache-aware load balancing, and a YAML-based inference policy engine. The post reports benchmarks in which Kinference cut median time-to-first-token by 95% and median end-to-end latency by 45% versus standard Kubernetes load balancing on one workload, and served 7.4× the goodput on a shared-prefix workload and about 3.5× on multiturn. It also covers per-tenant inference metrics and Helm-based installation steps.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 9Nov 7 19:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.