Hacker News

How to align with user preference in a RAG system?

Hacker News - Wed, 04/16/2025 - 1:18pm

Current embedding-based RAG systems primarily rely on semantic similarity. Given a document and a query, the system usually retrieves multiple sections that appear semantically relevant. However, in domain-specific applications, such as financial analysis or legal research, users often have domain-specific preferences for which parts of a document to consult first. These preferences are typically driven by experience about where answers are typically found or which sections are considered more trustworthy sources of information.

For example:

- When querying about financial performance metrics (e.g., earnings adjustments), experienced analysts typically look first at the Management’s Discussion and Analysis (MD&A) section or related financial statement footnotes.

- For questions about company risks, they usually prioritize the Risk Factors section before turning to broader disclosures.

These expert-driven navigation patterns are difficult to capture using embedding-based RAG alone. Fine-tuning embedding models to reflect such preferences is possible, but it tends to be costly and resource-intensive.

An alternative approach is to incorporate reasoning-based retrieval, which mimics how humans find information. For example, when reading a long document, a human typically starts by reviewing the table of contents to determine which sections to read first, based on the context of the query and preference. Similarly, one can build an LLM agent that analyzes the "table of contents" and then navigates through the document according to expert preferences. This can be achieved by using few-shot prompting, where the system learns from sample user preference examples provided in the prompt, allowing it to prioritize sections based on the user’s needs.

To support this paradigm, we developed an open-sourced tool called PageIndex. It can transform any long documents into an LLM-friendly "table-of-contents" tree index, which is ready for the LLM agents to navigate. With PageIndex, you can easily build RAG agents that align with user preferences and domain logic.

Would love any feedback, particularly thoughts on reasoning-based RAG or other potential applications of PageIndex.

Comments URL: https://news.ycombinator.com/item?id=43707928

Points: 7

# Comments: 1

Categories: Hacker News

Birth of Basic [video]

Hacker News - Wed, 04/16/2025 - 1:15pm
Categories: Hacker News

Show HN: Interview Terminator

Hacker News - Wed, 04/16/2025 - 1:15pm

Article URL: https://interm.ai/

Comments URL: https://news.ycombinator.com/item?id=43707884

Points: 1

# Comments: 0

Categories: Hacker News

Ask HN: Handling new director who doesn't seem great?

Hacker News - Wed, 04/16/2025 - 1:14pm

Hi,

About two months ago my company hired a new director, my skip manager.

A lot of things are off about him, IMO:

- he hasn't met the engineers on the team except for his three teams' leads, including me. - he worked in the same broader area, but in a different domain, and is insistent on applying things that worked in that other domain to this company. - he's top-down and doesn't know much about the facts on the ground. - he gives inconsistent information and direction to me and my direct manager. - he's introducing processes that aren't necessary. - he doesn't ask questions about the platform. - he's extremely focused on one particular aspect of the platform but doesn't know anything about the other goals of the platform - he second-guesses our hiring decisions before we make an offer; in one case, he re-interviewed a candidate we had approved of; in another, he was skeptical about an internal candidate.

Normally I'd give a new director a lot of leeway since they're still gathering context and information, and they were approved by my org's leadership in interviews. But enough is odd that I don't know if I'm going about things the best way.

So far I've attempted to extend our 1:1s to try to broaden his concerns to other parts of the platform, and to show the span of work we could do is much larger, and his suggestions aren't necessarily the best things we can work on, or at least should be contingent on doing some diligence before acting on them. That works to some extent. I thought it might be that he came in with some amount of distrust for me and this team -- that still might be the case, but it's clear that among his three teams, mine is the least problematic, at least right now.

But enough things smell wrong that I don't know if I should be doing something else, like giving him direct feedback, especially about being curious and orienting him towards being more bottom-up, or even going above his head.

Anyone have experience with a situation like this?

Comments URL: https://news.ycombinator.com/item?id=43707878

Points: 1

# Comments: 0

Categories: Hacker News

Show HN: Ivy Lee method for Slack teams

Hacker News - Wed, 04/16/2025 - 1:05pm

Article URL: https://tryivy.app/

Comments URL: https://news.ycombinator.com/item?id=43707769

Points: 1

# Comments: 0

Categories: Hacker News

Pages