14. Data engineering for RAG and AI applications

A retrieval-augmented generation (RAG) system is a search engine whose reader is a language model. Its answer quality is decided mainly by the data pipeline behind it: parsing, chunking, embedding, indexing and freshness. The prompt matters far less than most people assume.

Two pipelines, not one

• The indexing pipeline runs offline or continuously: ingest sources, parse, clean, chunk, enrich with metadata, embed, and write to an index.
• The query pipeline runs per request: take the question, optionally rewrite it, retrieve candidates, rerank them, assemble the context, generate the answer, and log everything.

Most RAG failures trace back to the indexing pipeline. If the right passage was never extracted cleanly, no prompt can recover it.

[embedded content: RAG system · indexing and query pipelines meeting at the index]

The query pipeline can only find what the indexing pipeline put into the index, so most quality work happens in the top row.

Ingestion and parsing

• Sources: document stores, wikis, ticket systems, PDFs, databases and APIs, each needing a connector.
• Permissions travel with data. Record who may see each document at ingestion time, as metadata. You will filter on it at query time.
• Change detection: store a content hash per document; re-process only what changed. Use CDC or webhooks where sources support them, and handle deletions explicitly.
• Parsing is the hardest step. PDFs carry layout, multi-column text, tables, headers, footers and scanned pages. Layout-aware parsers such as Unstructured, Docling and Marker, or cloud document AI services, recover structure that naive text extraction destroys.
• Cleaning: strip boilerplate, normalize Unicode and whitespace, and keep structure (headings, lists, tables) as Markdown where possible.

Chunking

You chunk because embedding models have input limits, because a small passage gives a more precise embedding than a whole document, and because the model’s context window is a budget.

Strategy	How	Strength	Weakness
Fixed size with overlap	Every N tokens, overlapping by M	Simple, predictable	Cuts mid-sentence and mid-table
Recursive	Split on headings, then paragraphs, then sentences until small enough	Respects natural boundaries	Uneven chunk sizes
Structure-aware	Follow Markdown sections, HTML elements, or code functions	Chunks are coherent units	Needs parsed structure
Semantic	Split where embedding similarity between sentences drops	Topic-coherent chunks	Slower, less predictable
Parent-child	Embed small chunks, return their larger parent section	Precise match, rich context	More storage and logic
Contextual	Prepend a short document-level summary to each chunk before embedding	Chunks keep their meaning out of context	Extra LLM cost at indexing time

The core trade-off: small chunks retrieve precisely but lose context; large chunks keep context but blur the embedding. A few hundred tokens with modest overlap is a reasonable starting point; then measure, because the right size depends on your documents and questions.

Metadata enrichment

Attach title, section path, source URL, date, author, language, document type and access groups to every chunk. Metadata enables filtering (“only 2026 policies”), citations, freshness rules and permission checks. It is cheap to store and expensive to add later.

Embeddings

An embedding model maps text to a vector so that texts with similar meaning land close together. Closeness is measured by cosine similarity, or by dot product on normalized vectors.

• Choosing a model: language coverage (German and multilingual matter for you), domain fit, vector dimension, maximum input length, cost and latency. The MTEB leaderboard is a starting point, but test on your own data.
• Vectors from different models are incompatible. Changing the embedding model means re-embedding the whole corpus. That is why the chunk table in section 5 records embedding_model.
• Size: Matryoshka-trained models let you truncate vectors to fewer dimensions with modest loss. Quantizing to float16, int8 or even binary shrinks memory several-fold.

Vector indexes: approximate nearest neighbour search

Exact search compares the query with every vector, which is too slow at millions of vectors. Approximate nearest neighbour (ANN) indexes trade a little recall for large speed gains.

Index	Idea	Trade-off
HNSW	A layered graph; search hops from coarse to fine neighbours	Fast and accurate, but memory-hungry; tuned with M, ef_construction, ef_search
IVF	Cluster vectors around centroids; search only the nprobe nearest clusters	Less memory, recall depends on nprobe
Product quantization (PQ)	Compress each vector into short codes	Large memory savings, some accuracy loss; often combined with IVF
DiskANN-style	Graph index designed to live on SSD	Scales beyond RAM with good latency

Always measure recall@k of the ANN index against exact search on a sample, alongside latency. Watch metadata filtering: filtering after the ANN search can leave too few results, so prefer engines that filter during the search.

Stores and libraries: pgvector, Qdrant, Weaviate, Milvus, Pinecone, Elasticsearch and OpenSearch, LanceDB and Chroma; FAISS is the classic library underneath many of them.

Hybrid search and reranking

Dense embeddings capture meaning but are weak on exact tokens: product codes, error IDs, names, legal paragraph numbers. Keyword search with BM25 over an inverted index is strong on exactly those. Hybrid search runs both and merges the rankings, commonly with reciprocal rank fusion, where each document’s score sums over the ranked lists it appears in:

k is a constant, often 60, that dampens the influence of top ranks. Then a reranker, a cross-encoder that reads the query and each candidate together, re-scores the top 50 to 100 candidates and keeps the best handful. Reranking is often the single largest quality gain in a RAG system.

Freshness and updates

• Re-embed only chunks whose content hash changed.
• Propagate deletions: a deleted source document must disappear from the index, or the system will cite documents that no longer exist.
• For model changes, build a new index alongside the old one and switch an alias when it is ready, so search never goes down.

Permissions

Filter retrieval by the requesting user’s access groups at query time, before anything reaches the model. Never rely on instructions to the LLM to withhold content it has already been given.

Evaluating retrieval and generation

Build a test set of questions with known relevant documents and reference answers, drawn from real user questions where possible. Then evaluate each stage separately:

• Retrieval: recall@k (did the right chunk appear in the top k?), precision@k, MRR and nDCG.
• Generation: faithfulness (is every claim supported by the retrieved context?), answer relevance and correctness.
• LLM-as-judge: scalable, but calibrate it against human judgements and version the judge.

If you only measure final answers, you cannot tell whether a failure came from parsing, chunking, retrieval or generation.

Logging, tracing and feedback

For every request, record the query, rewritten query, retrieved chunk IDs and scores, prompt version, model, output, token counts, latency, cost and any user feedback. OpenTelemetry has emerging conventions for generative AI traces, and tools such as Langfuse, LangSmith and Arize Phoenix store and display them.

These traces are your next evaluation set and fine-tuning dataset. They also contain user data, so apply retention limits and PII handling from day one.

Agents and memory

Agents read and write data through tools, which turns every tool into a data interface with schemas, permissions and failure modes. Agent “memory” is a data store with the usual questions: what is written, how it is retrieved, how long it is kept, and who may see it.

Questions to ask

• When a wrong answer happens, can I see exactly which chunks were retrieved and why?
• What is our retrieval recall@k on a labeled test set, and how did the last change move it?
• How long after a source document changes does the index reflect it, deletions included?
• Does retrieval enforce the user’s permissions before the model sees anything?
• Which embedding model produced each vector, and what would switching cost?