Hacker News

Subscribe to Hacker News feed
Hacker News RSS
Updated: 27 min 21 sec ago

Identity Access is not just for technical people

46 min 26 sec ago

Article URL: https://pathros.cc/local/

Comments URL: https://news.ycombinator.com/item?id=48406634

Points: 1

# Comments: 1

Categories: Hacker News

Show HN: LLMhop – A tiny, stateless router for LLMs with a NixOS module

1 hour 3 min ago

LLMhop is a tiny stateless proxy for LLM inference servers. It tackles an issue I faced when trying to serve more than one local LLM at once which is not natively supported by vLLM. The LLMhop binary inspects the model field of the request and routes it to the correct backend service with optional handling of authentication. In addition, it contains a NixOS module to run llama.cpp, vLLM, and sglang via Quadlet/Podman and auto-register with the proxy.

Comments URL: https://news.ycombinator.com/item?id=48406517

Points: 1

# Comments: 0

Categories: Hacker News

Pages