Hacker News

Hi HN! This is Adil, Salman, Co and Shuguang and we're excited to introduce archgw [1], an open source intelligent proxy for agents built on Envoy [2]. Arch moves the critical but crufty work around safety, observability, and routing of prompts outside business logic. Arch is a uniquely intelligent infrastructure primitive, engineered with purpose-built fast LLMs [3] for tasks like intent detection over multi-turn, parameter identification and extraction, triggering single/multiple function calls, and offers convenience features to auto dispatch LLM calls for summarization based on data from your APIs via system prompts configured in archgw.

Today, the approach to build a smart production-ready agent is weaving together a large set of mono-functional opinionated libraries, adding extra layers like LLM-based preprocessing to determine things like relevance and safety of the user's prompt (e.g. applying governance and guardrails). Once past that stage, developers must extract relevant information from the user prompt to determine intent, extract parameters as necessary, package relevant tools calls to an LLM to trigger a backend API to execute particular domain-specific task. etc. After all that is done then only are developers ready to trigger an LLM call for summarization and must manage upstream error handling and retry logic themselves. Not to mention, if they want to experiment with multiple LLMs or move between LLM versions, they have to write crufty undifferentiated code. This entire experience is slow, error prone, cumbersome, and not specifically unique.

Prior to building archgw, the team spent time building Envoy [2] at Lyft, API Gateway at AWS, specialized search and intent models at Microsoft Research and worked on safety at Meta. archgw was born out of the belief that several rules based mono-functional tools should be converged into a multi-functional infrastructure primitive designed for prompts and agents. We built archgw on the highly popular, battle-tested open source proxy Envoy and re-imagined it for prompts and agents. For this we had to build blazing fast LLMs [3] that can handle crufty, ahead-in-the-request-path type of work in handling and processing prompts that are sent to an agent, so that developers can focus on what matters most: building fast personalized agents without the unnecessary prompt engineering and systems integration work needed to get there.

Here are some additional details about the open source project. arghw is written in rust, and the request path has three main parts:

* Listener subsystem which handles downstream (ingress) and upstream (egress) request processing.

* Prompt handler subsystem. This is where archgw makes decisions on the safety of the incoming request via its prompt_guard primitive and identifies where to forward the conversation to via its prompt_target primitive.

* Model serving subsystem is the interface that hosts all the lightweight LLMs engineered in archgw and offers a framework for things like hallucination detection of our these models

We loved building this open source project, and our belief is that this infra primitive would help developers build faster, safer and more personalized agents without all the manual prompt engineering and systems integration work needed to get there. We hope to invite other developers to use and improve Arch. Please give it a shot and leave feedback here, or at our discord channel [4]

Also here is a quick demo of the project in action [5]. You can check out our public docs here at [6]. Our models are also available here [7].

[1] https://github.com/katanemo/archgw

[2] https://www.envoyproxy.io/

[3] https://huggingface.co/collections/katanemo/arch-function-66...

[4] https://discord.com/channels/1292630766827737088/12926307682...

[5] https://www.youtube.com/watch?v=I4Lbhr-NNXk

[6] https://docs.archgw.com/

[7] https://huggingface.co/katanemo

Comments URL: https://news.ycombinator.com/item?id=42187132

Points: 8

# Comments: 3

Article URL: https://venturebeat.com/ai/microsoft-quietly-assembles-the-largest-ai-agent-ecosystem-and-no-one-else-is-close/

Comments URL: https://news.ycombinator.com/item?id=42186837

Points: 1

# Comments: 0

Article URL: https://www.wsj.com/tech/ai/ai-college-costs-higher-education-9a8f875d

Comments URL: https://news.ycombinator.com/item?id=42186833

Points: 1

# Comments: 0

Article URL: https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf

Comments URL: https://news.ycombinator.com/item?id=42186819

Points: 1

# Comments: 0

Article URL: https://www.bloomberg.com/opinion/articles/2024-11-19/insider-trading-at-the-fed

Comments URL: https://news.ycombinator.com/item?id=42186811

Points: 1

# Comments: 0

Article URL: https://arstechnica.com/tech-policy/2024/11/cable-companies-and-trumps-fcc-chair-agree-data-caps-are-good-for-you/

Comments URL: https://news.ycombinator.com/item?id=42186802

Points: 1

# Comments: 0

Article URL: https://www.microsoft.com/en-us/windows-365/link

Comments URL: https://news.ycombinator.com/item?id=42186796

Points: 2

# Comments: 0

Article URL: https://twitter.com/cyrilzakka/status/1858937683487412704

Comments URL: https://news.ycombinator.com/item?id=42186794

Points: 1

# Comments: 0

Article URL: https://www.theguardian.com/world/2024/nov/19/bird-flu-cases-mutation-canada

Comments URL: https://news.ycombinator.com/item?id=42186777

Points: 2

# Comments: 0

Article URL: https://techcommunity.microsoft.com/blog/windows-itpro-blog/windows-365-link%E2%80%94the-first-cloud-pc-device-for-windows-365/4302687

Comments URL: https://news.ycombinator.com/item?id=42186773

Points: 1

# Comments: 1

It's been a few weeks now since Claude Computer Use was launched, and OpenAI has been tipped to release something similar by the end of the year.

Curious if anyone has been using it, and if so, for what?

Comments URL: https://news.ycombinator.com/item?id=42186768

Points: 2

# Comments: 0

Article URL: https://zverok.space/blog/2021-03-16-spellchecking-dictionaries.html

Comments URL: https://news.ycombinator.com/item?id=42186761

Points: 1

# Comments: 0

Article URL: https://www.nature.com/articles/d41586-024-03746-y

Comments URL: https://news.ycombinator.com/item?id=42186751

Points: 1

# Comments: 0

Article URL: https://now.tufts.edu/2024/11/07/people-schizophrenia-show-distinct-brain-activity-when-faced-conflicting-information

Comments URL: https://news.ycombinator.com/item?id=42186750

Points: 1

# Comments: 0

Article URL: https://userinyerface.com/game.html

Comments URL: https://news.ycombinator.com/item?id=42186747

Points: 1

# Comments: 0

Article URL: https://neal.fun/deep-sea/

Comments URL: https://news.ycombinator.com/item?id=42186746

Points: 2

# Comments: 0

Article URL: https://news.mit.edu/2024/emvolon-turns-automotive-engines-into-green-fuel-chemical-plants-1119

Comments URL: https://news.ycombinator.com/item?id=42186729

Points: 1

# Comments: 0

Article URL: https://www.seroundtable.com/google-analytics-missing-data-nov-13-38432.html

Comments URL: https://news.ycombinator.com/item?id=42186726

Points: 1

# Comments: 0

Article URL: https://metricool.com/tiktok-study/

Comments URL: https://news.ycombinator.com/item?id=42186723

Points: 1

# Comments: 0

Article URL: https://github.com/charbz/gophers

Comments URL: https://news.ycombinator.com/item?id=42186685

Points: 1

# Comments: 0