Data Engineering for AI Systems from First PrinciplesPost 18 of 21 · 21 of 21 written
  1. How to use this guide
  2. 01Why AI runs on data engineering
  3. 02Representing data: bytes, types, schemas, formats
  4. 03Storage: where bytes live and what reading them costs
  5. 04Databases from first principles
  6. 05Data modeling: shaping data for its readers
  7. 06Distributed systems: the physics of many machines
  8. 07Batch processing: transforming data in bulk
  9. 08Streaming: data that never ends
  10. 09Architectures: warehouse, lake, lakehouse
  11. 10Pipelines and orchestration
  12. 11Data quality, contracts, lineage and observability
  13. 12Data engineering for machine learning
  14. 13Data engineering for LLMs
  15. 14Data engineering for RAG and AI applications
  16. 15Feeding GPUs: data loading for training
  17. 16Governance, privacy and security
  18. 17Cost and performance reasoning
  19. 18The tool landscape, mapped to principles
  20. 19Learning path and hands-on projects
  21. 20Question bank for fluency
  22. 21Glossary

18. The tool landscape, mapped to principles

Data Engineering · 18 of 213 min readOct 2026

Every tool answers one or two of the six questions from section 0, and makes a specific trade-off. Place a new tool on this map first; its details then become easy to learn. Names below are representative as of late 2026, not endorsements.

Layer Answers Open source examples Managed / commercial examples See
Object storage Storage MinIO, Ceph Amazon S3, Google Cloud Storage, Azure Blob 3
File formats Representation Parquet, Avro, Arrow, Lance - 2
Table formats Storage, trust Iceberg, Delta Lake, Hudi, Paimon Built into Databricks, Snowflake, BigQuery 9
Catalogs Trust Unity Catalog OSS, Apache Polaris, Nessie, Hive Metastore AWS Glue, Databricks Unity Catalog 9
Operational databases Storage, serving PostgreSQL, MySQL Amazon RDS, Aurora, Cloud SQL, Neon, Supabase 4
Warehouses and query engines Transformation, serving DuckDB, ClickHouse, Trino Snowflake, BigQuery, Redshift, Databricks SQL, MotherDuck 4, 7
Batch compute Transformation Spark, Polars, Ray, Dask Databricks, EMR, Dataproc, Anyscale 7
Event transport Movement Kafka, Redpanda, Pulsar Confluent Cloud, Kinesis, Pub/Sub 8
Stream processing Transformation Flink, Spark Structured Streaming, Kafka Streams, RisingWave Confluent Flink, Materialize 8
Change data capture Movement Debezium Fivetran, cloud database streams 8
Ingestion Movement Airbyte, dlt, Meltano Fivetran 10
SQL transformation Transformation dbt Core, SQLMesh dbt Cloud 10
Orchestration Movement, trust Airflow, Dagster, Prefect, Kubeflow Pipelines, Flyte, Temporal Astronomer, Cloud Composer, Dagster+ 10
Data quality Trust Great Expectations, Soda Core, Pandera, Deequ Monte Carlo, Soda Cloud 11
Lineage and metadata Trust OpenLineage, DataHub, OpenMetadata, Marquez Atlan, Collibra 11
Feature stores Serving Feast, Hopsworks Tecton, Databricks, Vertex AI, SageMaker 12
Experiment tracking and registry Trust MLflow Weights & Biases 12
Data versioning Trust DVC, lakeFS Hugging Face Hub 12
Labeling Transformation Label Studio, Argilla, CVAT Scale AI, Labelbox 12
LLM data curation Transformation datatrove, NeMo Curator, Dolma toolkit - 13
Training data loading Serving PyTorch DataLoader, WebDataset, MosaicML Streaming, NVIDIA DALI - 15
Document parsing Representation Unstructured, Docling, Marker Cloud document AI services, LlamaParse 14
Vector search Serving pgvector, Qdrant, Weaviate, Milvus, LanceDB, Chroma, FAISS Pinecone, managed versions of the open tools 14
Keyword search Serving OpenSearch, Elasticsearch Elastic Cloud, managed OpenSearch 14
LLM tracing and evaluation Trust Langfuse, Arize Phoenix, Ragas LangSmith, Braintrust 14
Infrastructure All Docker, Kubernetes, Terraform, OpenTofu Every cloud’s managed versions 10

How to evaluate any new tool in ten minutes

  1. Which of the six questions does it answer?
  2. What trade-off does it make: latency, throughput, cost, consistency or simplicity?
  3. Does it store data in an open format, or lock it in?
  4. What happens when it fails mid-operation: duplicates, partial writes or nothing?
  5. Who operates it, and what does that cost in people-hours?
  6. How is it priced, and how does price scale with data volume?
  7. What does it replace, and is that thing actually a problem for you today?

A sensible stack for a solo AI engineer

You do not need most of the table to build real AI systems. A small, durable starting stack:

  • PostgreSQL with pgvector for application data, metadata and vectors up to millions of chunks.
  • Parquet files in object storage (or MinIO locally) as the raw and processed data lake.
  • DuckDB and Polars for transformation and analysis on one machine.
  • dbt or plain versioned SQL for repeatable transformations.
  • Dagster or Prefect once you have more than a few scheduled jobs.
  • Pandera or Pydantic for validation, at every boundary.
  • MLflow to link datasets, runs and models; Langfuse to trace LLM calls.

Add Spark, Kafka, a lakehouse table format or a dedicated vector database only when you can name the specific limit you have hit.