Hacker News AI· avelone·· 2 hr agoAI score58
Show HN: Shard – Distributed AI Inference Across Multiple Computers
Show HN: Shard – Distributed AI Inference Across Multiple Computers
AI summary
SHARD (shard-engine) is an open-source distributed local LLM inference engine written in Python with FastAPI and PyTorch, a GGUF/llama.cpp backend, and native C++20/MASM AVX2 SIMD kernels. It pools GPU VRAM and system RAM across LAN machines via UDP discovery on port 8001 and exposes OpenAI- and Anthropic-compatible endpoints on port 8000, with a `shard fit` command to check model fit before downloading. The project is at v1.8.1 under the MIT License and installs via PyPI, source, or Docker Compose.
Source: Hacker News AI · github.com