Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes
Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes
Buoyant announced Kinference, an Apache v2 open source control plane for building inference platforms on vLLM and Kubernetes, offering token rate limiting, tenant usage and cost tracking, KV-cache-aware load balancing, and a YAML-based inference policy engine. The post reports benchmarks in which Kinference cut median time-to-first-token by 95% and median end-to-end latency by 45% versus standard Kubernetes load balancing on one workload, and served 7.4× the goodput on a shared-prefix workload and about 3.5× on multiturn. It also covers per-tenant inference metrics and Helm-based installation steps.
It gives a concrete look at cache-aware routing and policy-driven model selection for self-hosted vLLM deployments, backed by published benchmarks.
Source: vLLM Blog · vllm.ai