iCreatAI News
Skip to content
Hacker News AI· Anh_khoa·· 7 hr agoAI score34

Show HN: Vantage-V-AI – 2-bit ternary KV-cache with attention sinks

Show HN: Vantage-V-AI – 2-bit ternary KV-cache with attention sinks

AI summary

Vantage-V-AI (v8.2.0) is an open-source, bounded-memory inference runtime that integrates with Hugging Face AutoModelForCausalLM via a one-line vantage.enable() call to cap the active attention window and retain sink tokens. It uses a 2-bit packed ternary KV-cache (4 elements per uint8 byte, 7.97x storage reduction over FP16) plus attention sinks, and reports benchmarks on 2x NVIDIA Tesla T4 GPUs with Qwen2.5-3B-Instruct, including 100.0% exact token parity between lossless and FP8 streaming modes and 0.988 average retrieval coherence from 512 to 4,096 tokens. The repository is at github.com/khoadev-vn/VANTAGE-V-AI under the MIT License.

Source: Hacker News AI · github.com