How to Build a RAG Pipeline from Scratch

Retrieval-Augmented Generation (RAG) is how you make LLMs answer questions about your data instead of hallucinating. It's the most common pattern in production AI, and also the most misunderstood.

This guide walks through every step, with the trade-offs nobody talks about.

What RAG Actually Does

Instead of asking an LLM to answer from memory (which leads to hallucinations), RAG retrieves relevant documents first, then feeds them as context to the LLM. The model generates an answer grounded in your actual data.

User question
  → Step 1: Embed the question into a vector
  → Step 2: Search your vector store for similar document chunks
  → Step 3: Feed retrieved chunks + question to an LLM
  → Step 4: LLM generates an answer grounded in the retrieved context
Output: answer + source citations

Simple concept. The devil is in each step.

Step 1: Document Ingestion

Before anything, you need your data in a searchable format.

Common sources: PDFs, web pages, Notion docs, Google Docs, Confluence, databases, Slack messages.

Key decision: Extract text or use OCR?

• Text-based PDFs → direct extraction (fast, accurate)
• Scanned documents or images → OCR (slower, error-prone)
• Mixed documents → detect and route accordingly

Watch out for: Tables (most extractors mangle them), headers/footers (pollute search results), and multi-column layouts (break reading order).

Step 2: Chunking

You can't feed entire documents to an LLM because they're too long and most of the content is irrelevant to any given question. Chunking splits documents into smaller, searchable pieces.

Common strategies:

The critical trade-off: Smaller chunks = more precise retrieval but less context per chunk. Larger chunks = more context but noisier retrieval. Most production systems land at 512 tokens with 50-token overlap.

Overlap matters. Without overlap, a relevant sentence split across two chunks becomes invisible to search. 10-15% overlap catches these edge cases.

Step 3: Embeddings

Convert each chunk into a numerical vector that captures its semantic meaning. Similar chunks produce similar vectors.

Model choices in 2026:

Key decision: Cloud API or on-premise?

• Cloud APIs are simpler but cost scales with volume
• On-premise is free per-query but requires GPU infrastructure
• Hybrid: on-premise for bulk ingestion, cloud API for real-time queries

Step 4: Vector Storage

Store embeddings in a database optimized for similarity search.

Popular options:

Our recommendation: pgvector. If you're already running PostgreSQL (and most teams are), adding pgvector means no new infrastructure, no new vendor, and your vectors live next to your relational data. For most production workloads under 10M vectors, pgvector with HNSW indexing performs well.

Step 5: Retrieval (The Hard Part)

This is where most RAG pipelines fail. Pure vector search has blind spots.

The problem with vector-only search:

• Semantic search finds conceptually similar content but misses exact terms
• "What is the API rate limit?" might retrieve docs about "usage quotas" but miss the page that literally says "rate limit: 100 req/s"

The solution: Hybrid search. Combine vector similarity (semantic) with BM25 keyword matching (lexical). This catches both conceptual matches and exact term matches.

Score = α × cosine_similarity + (1 - α) × BM25_score

Where α typically ranges from 0.5 to 0.7 depending on your data.

Retrieval count matters. Retrieve too few chunks (k=2) and you miss relevant context. Retrieve too many (k=20) and you dilute the signal with noise. k=5 to k=8 works well for most use cases. Some systems use a two-stage approach: retrieve k=20 with vector search, then rerank to top 5 using a cross-encoder.

Step 6: LLM Generation

Feed the retrieved chunks as context to an LLM with a carefully constructed prompt.

Prompt structure:

System: You are a helpful assistant. Answer questions based ONLY
on the provided context. If the context doesn't contain the answer,
say "I don't have enough information to answer that."

Context:
{retrieved_chunks}

User: {original_question}

Key decisions:

• Model selection. GPT-4 for complex reasoning, Llama-3/Mistral for speed and cost. Route based on query complexity.
• Temperature. Use 0 or 0.1 for factual Q&A. Higher temperatures increase creativity but also hallucination risk.
• Citation. Ask the model to cite which chunks it used. This makes answers verifiable and builds user trust.

Common RAG Failures (and Fixes)

Building RAG in Akaitto

Akaitto has the RAG workflow surface built in:

• Document ingestion with automatic format detection (PDF, web, docs)
• Smart chunking with configurable strategies and overlap
• Hybrid search combining pgvector cosine similarity with BM25 keyword matching
• Per-step cost and latency tracking so you know exactly where time and money go
• Execution traces showing route decisions, retrieval steps, retries, and output status

Instead of stitching together 5 different libraries and managing the infrastructure yourself, you get a production-ready RAG workflow with traces and cost visibility from day one.

The best RAG pipeline is the one you can actually monitor and optimize.