Hacker News

Show HN: LLMhop – A tiny, stateless router for LLMs with a NixOS module

Hacker News - 2 hours 32 min ago

LLMhop is a tiny stateless proxy for LLM inference servers. It tackles an issue I faced when trying to serve more than one local LLM at once which is not natively supported by vLLM. The LLMhop binary inspects the model field of the request and routes it to the correct backend service with optional handling of authentication. In addition, it contains a NixOS module to run llama.cpp, vLLM, and sglang via Quadlet/Podman and auto-register with the proxy.

Comments URL: https://news.ycombinator.com/item?id=48406517

Points: 1

# Comments: 0

Categories: Hacker News

Pages