Putting AI on your own documents without it making things up

Once a business wants an assistant that answers from its own documents — policies, product sheets, past tickets — you are at Stage 2 of the NGAZI ladder. The technique that gets you there is retrieval: you find the relevant text first, then let the model answer using only that.

This post is for engineers. If you just want the business view, start with where to actually begin.

The shape of the problem#

A base model knows a lot about the world and nothing about your business. Asked about your refund policy, it will produce something that sounds like a refund policy — fluent, plausible, and not yours. That is the failure we are designing against.

Retrieval-augmented generation (RAG) fixes this by changing the question. Instead of "answer this," we ask "answer this using only the text below," and we make sure the text below actually contains the answer.

The minimal pipeline#

Four steps: chunk your documents, embed them, retrieve the closest chunks to a question, and generate an answer constrained to those chunks.

from anthropic import Anthropic
 
client = Anthropic()
 
def answer(question: str, store) -> str:
    # 1. Retrieve the most relevant chunks for this question.
    chunks = store.search(question, k=5)
    context = "\n\n---\n\n".join(c.text for c in chunks)
 
    # 2. Generate, constrained to the retrieved context.
    message = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system=(
            "Answer using ONLY the context provided. "
            "If the context does not contain the answer, say you don't know. "
            "Never guess a fact that isn't in the context."
        ),
        messages=[
            {
                "role": "user",
                "content": f"<context>\n{context}\n</context>\n\nQuestion: {question}",
            }
        ],
    )
    return message.content[0].text

The whole game is in store.search. Get the right five chunks and a capable model answers correctly almost for free. Retrieve the wrong chunks and no amount of prompt engineering saves you.

Chunking is a real decision#

"Split the document" hides a lot of choices. Chunk too large and you dilute the signal and blow your token budget; too small and you sever the sentence that gave a fact its meaning.

For most business documents, splitting on headings with a small overlap gets you 80% of the way. Reach for semantic chunking when recall on messy content is the thing standing between you and shipping.

Evaluate the two halves separately#

The most common debugging mistake is treating RAG as one box. It is two. When an answer is wrong, ask first: were the right chunks retrieved?

• If the answer was not in the retrieved context, it is a retrieval bug — fix chunking, embeddings, or k.
• If the answer was in the context but the model still got it wrong, it is a generation bug — fix the prompt or the model.

question ──▶ [ retriever ] ──▶ chunks ──▶ [ generator ] ──▶ answer
                  ▲                              ▲
             evaluate here                  and here

Keep a small set of real questions with known answers and known source chunks. Score retrieval (did the right chunk come back?) and generation (given the right chunk, was the answer correct?) as separate numbers. You will fix problems in half the time.

Where to go next#

Once retrieval is solid, the next rung — Stage 3, AI inside the workflow — is about letting the assistant act, not just answer. That raises the stakes and the guardrails. The NGAZI framework lays out what changes and why you should not skip a rung to get there.