Glossary · Data Engineering

Term	Meaning
ACID	Atomicity, consistency, isolation, durability: the guarantees of a database transaction
ANN	Approximate nearest neighbour search: fast vector search that trades a little recall for speed
Arrow	A standard in-memory columnar layout that lets tools share data without copying
As-of join	A join that takes, for each row, the latest matching value at or before that row’s timestamp
Backfill	Re-running a pipeline over past partitions
Backpressure	Slowing a producer when its consumer cannot keep up
BM25	A classic keyword relevance score used by search engines
Bronze, silver, gold	The medallion layers: raw, cleaned, use-case ready
B-tree	A balanced sorted tree index, good for lookups and range scans
CAP / PACELC	Trade-offs between consistency, availability and latency in distributed stores
CDC	Change data capture: streaming a database’s inserts, updates and deletes from its log
Chunk	A passage of a document, embedded and retrieved as one unit in RAG
Compaction	Merging many small files or sorted runs into fewer large ones
Concept drift	A change in the relationship between inputs and the target
Consumer group	A set of consumers sharing a topic’s partitions in Kafka
Contamination	Evaluation data leaking into training data
Copy-on-write / merge-on-read	Two ways a table format applies updates: rewrite files, or record changes and merge when reading
DAG	Directed acyclic graph: the dependency structure of a pipeline
Data contract	An explicit agreement on schema, meaning and guarantees between producer and consumer
Data drift	A change in the distribution of model inputs
Data interval	The period of data an orchestrated run is responsible for
Deduplication	Removing exact, near or semantic duplicates
Dimension table	Context about entities (who, what, where) in a dimensional model
ELT / ETL	Load then transform, or transform then load
Embedding	A vector representing meaning, so similar items lie close together
Event time	When an event actually happened, as opposed to when it was processed
Fact table	Measurable events in a dimensional model
Feature store	A system serving consistent features for training (offline) and inference (online)
Grain	What exactly one row of a table represents
HNSW	A layered graph index for fast vector search
Hybrid search	Combining keyword and vector retrieval
Idempotency	The property that running an operation twice equals running it once
Inverted index	A map from each term to the documents containing it
IVF / PQ	Vector index techniques: cluster-based search, and vector compression into codes
Jaccard similarity	Size of the intersection of two sets divided by the size of their union
Lakehouse	Open files in object storage plus a table format providing transactions and schemas
Late data	Events that arrive after the system considered their window complete
Leakage	Information unavailable at prediction time reaching the training data
Lineage	The graph of how datasets derive from one another
Log (append-only)	An ordered, replayable sequence of records with offsets
LSH	Locality-sensitive hashing: finds likely-similar pairs without comparing all pairs
LSM-tree	A write-optimized index of sorted immutable runs, merged in the background
MinHash	A compact signature whose agreement estimates Jaccard similarity
MVCC	Multi-version concurrency control: readers see consistent snapshots while writers proceed
Normalization	Storing each fact once to avoid redundancy and contradictions
Object storage	A flat key-to-blob store over HTTP, such as S3
OLTP / OLAP	Transactional workloads versus analytical workloads
Offset	A record’s position in a log partition
Packing	Concatenating documents to fill fixed-length training sequences
Parquet	An open columnar file format with embedded schema and statistics
Partition	A slice of data placed or processed separately, by key, hash or time
Point-in-time correctness	Using only information available at each example’s prediction time
Predicate pushdown	Applying filters at the storage layer to skip data early
Pseudonymization	Replacing identifiers with tokens; still personal data under the GDPR
RAG	Retrieval-augmented generation: retrieving context to ground a model’s answer
Recall@k	The share of relevant items that appear in the top k results
Reranker	A model that re-scores retrieved candidates by reading query and candidate together
RRF	Reciprocal rank fusion: merging ranked lists by summing 1 ÷ (k + rank)
Schema evolution	Changing a schema while keeping old and new readers and writers working
SCD Type 2	Recording dimension changes as new rows with validity dates, preserving history
Shard	One large file of many samples, read sequentially during training
Shuffle	Redistributing data across machines by key; the expensive part of distributed jobs
Skew	Uneven distribution of data or work across partitions
Snapshot	A consistent version of a table at a point in time
Spine	The base rows of a training set: entity, prediction time, label
Star schema	A fact table joined to surrounding dimension tables
Time travel	Querying a table as of an earlier snapshot
Training-serving skew	Differences between features at training time and at serving time
Vectorized execution	Processing batches of column values per CPU instruction loop
WAL	Write-ahead log: changes recorded sequentially before being applied
Watermark	A stream processor’s estimate that events before a time have all arrived
Window	A finite slice of a stream: tumbling, sliding or session
Write-audit-publish	Stage output, validate it, then publish it atomically