RAG Explained: How Custom AI Assistants Answer From Your Own Documents

RAG — retrieval-augmented generation — is the difference between asking a model what it remembers and asking it to read the relevant page first. The system searches your documents, passes the best passages to the model with the question, and asks for an answer grounded in those passages. Done well, every answer links back to the source.

RAG in one paragraph

Your documents are split into chunks and converted into embeddings — numeric representations of meaning. When someone asks a question, the question is embedded too, and the system retrieves the chunks whose meaning is closest. Those chunks become context for the language model, which writes an answer and cites the passages it used. Nothing is trained into the model, so updating a document updates the answer.

What happens inside a RAG build

1. Ingest. Pull documents from where they already live — a drive, a CMS, a help desk, a database — and record permissions so the assistant cannot surface a file the user is not allowed to read.
2. Parse and chunk. Convert PDFs, scans and web pages into clean text, then split it into passages that keep headings and tables intact. Bad chunking is the most common cause of bad answers.
3. Embed and index. Turn each chunk into a vector and store it with its source metadata. Re-index when documents change.
4. Retrieve. At question time, find the nearest chunks, usually with a hybrid of semantic and keyword search so exact names and codes are not missed.
5. Rerank. Score the candidates again for relevance to the specific question and keep only the strongest few. This step is cheap and improves precision noticeably.
6. Generate with citations. Ask the model to answer only from the supplied passages, cite each claim, and refuse when the passages do not contain the answer.

RAG vs fine-tuning

What separates a useful assistant from a demo

• Citations a person can check. Every claim points to a passage, page and document. “Trust me” is not an answer.
• A refusal path. If the documents do not cover the question, the assistant says so. This is tested with unanswerable questions, not assumed.
• Permissions. The retrieval layer respects who the user is. A support agent should not see an HR document because the embedding matched.
• Freshness. Ingestion runs on a schedule or a webhook, and stale documents are removed as well as added.
• An evaluation set. Forty to a hundred real questions with known good answers, run before release and after every change.

Where RAG pays off

• Contracts and policy manuals where the answer must be quoted accurately.
• Past proposals and RFPs, so a team starts from the strongest previous response instead of a blank page.
• Support knowledge bases, with escalation when confidence is low.
• Onboarding material, where new hires ask the same twenty questions for a month.
• Operational documents spread across drives, where nobody knows which version is current.

What it costs and how long it takes

A focused prototype over a small document set starts at USD 3,000 and can be ready in three to five weeks. A production assistant — permissions, integrations, evaluation, monitoring and a review step — typically runs USD 8,000–25,000 over six to ten weeks. The variable is rarely the model; it is document quality, access rules and how many systems the assistant has to touch.

Common failure modes

• Chunks that split a table or a clause in half, so the retrieved passage is technically relevant but practically useless.
• No reranking, so the model receives ten loosely related passages and picks the wrong one.
• Permissions bolted on after retrieval, exposing documents across teams.
• No evaluation set, so nobody can tell whether a prompt change helped or hurt.
• Indexing everything once and never again, which turns a current assistant into a historical archive.

If you want this built with an evaluation report rather than a demo video, see our AI development practice or the on-device assistant we shipped.