SHARD open-sourced: pooling multi-machine VRAM across LANs for inference
Understand this story
avelone released the open-source distributed local LLM inference engine SHARD (shard-engine), which uses UDP discovery and llama.cpp RPC to pool the GPU VRAM and system memory of multiple machines on a LAN into a single inference cluster, supporting GGUF and PyTorch models from 7B to 70B+. SHARD can be installed directly from PyPI; on first run, GGUF inference automatically downloads the llama-server / ggml-rpc-server platform binaries, with no additional ML packages required; CUDA PyTorch inference requires first installing a matching CUDA version of torch, then running pip install "shard-engine[full]". According to the release notes, distributed layered execution across heterogeneous LAN GPUs works best on low-latency wired Gigabit/2.5GbE Ethernet subnets, and Wi-Fi links may experience higher inter-token latency.
AI overview based on source reports · Updated 38 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIShow HN: Shard – Distributed AI Inference Across Multiple Computers
SHARD (shard-engine) is an open-source distributed local LLM inference engine written in Python with FastAPI and PyTorch, a GGUF/llama.cpp backend, and native C++20/MASM AVX2 SIMD kernels. It pools GPU VRAM and system RAM across LAN machines via UDP discovery on port 8001 and exposes OpenAI- and Anthropic-compatible endpoints on port 8000, with a `shard fit` command to check model fit before downloading. The project is at v1.8.1 under the MIT License and installs via PyPI, source, or Docker Compose.
Activity for this story
Not enough continuous observations to show a trend yet.