iCreatAI News
Skip to content
Trending storyUpdating

SHARD open-sourced: pooling multi-machine VRAM across LANs for inference

1 reports1 sources1 hr agoUpdated

Understand this story

AI overview

avelone released the open-source distributed local LLM inference engine SHARD (shard-engine), which uses UDP discovery and llama.cpp RPC to pool the GPU VRAM and system memory of multiple machines on a LAN into a single inference cluster, supporting GGUF and PyTorch models from 7B to 70B+. SHARD can be installed directly from PyPI; on first run, GGUF inference automatically downloads the llama-server / ggml-rpc-server platform binaries, with no additional ML packages required; CUDA PyTorch inference requires first installing a matching CUDA version of torch, then running pip install "shard-engine[full]". According to the release notes, distributed layered execution across heterogeneous LAN GPUs works best on low-latency wired Gigabit/2.5GbE Ethernet subnets, and Wi-Fi links may experience higher inter-token latency.

AI overview based on source reports · Updated 38 min ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 18
  1. Hacker News AI
    Show HN: Shard – Distributed AI Inference Across Multiple Computers

    SHARD (shard-engine) is an open-source distributed local LLM inference engine written in Python with FastAPI and PyTorch, a GGUF/llama.cpp backend, and native C++20/MASM AVX2 SIMD kernels. It pools GPU VRAM and system RAM across LAN machines via UDP discovery on port 8001 and exposes OpenAI- and Anthropic-compatible endpoints on port 8000, with a `shard fit` command to check model fit before downloading. The project is at v1.8.1 under the MIT License and installs via PyPI, source, or Docker Compose.

Activity for this story

Not enough continuous observations to show a trend yet.