10. Git for AI and ML work

Git is excellent for small text that changes line by line - code, configs, prompts - and poor at large binaries such as datasets and checkpoints. AI work therefore pairs Git with tools that keep big files elsewhere while Git versions small pointers to them.

What to version, and where

Why large binaries break Git

This follows straight from Part 2. Every version of every file is stored for good, and every clone downloads all reachable history. Commit a 2 GB checkpoint ten times and every clone can carry up to 20 GB, because binary files compress badly against each other.

Deleting the file in a later commit does not help. The old commits still reach it, so it stays. Only rewriting history removes it. Hosts also set hard limits; GitHub, for example, rejects files over 100 MB.

The pointer pattern

Every large-file tool uses the same idea, which is Part 2 again one layer up. Commit a small text pointer holding the file’s hash and size; store the real bytes in separate storage, addressed by that hash.

Git LFS plugs into Git through filters. On add, it swaps the content for a pointer; on checkout, it downloads the real file.

git lfs install
git lfs track "*.safetensors"     # writes the rule into .gitattributes
git add .gitattributes model.safetensors
git commit -m "Add fine-tuned weights"
git push

version https://git-lfs.github.com/spec/v1
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
size 1342177280

That pointer is what Git actually commits. The classic mistake is adding the file before running git lfs track, which puts the full binary into normal history. Hosts usually meter LFS storage and bandwidth.

DVC is a separate command-line tool built for data and pipelines. It writes small .dvc pointer files for Git and pushes the data to a remote you choose, such as S3, GCS or a shared drive.

pip install "dvc[s3]"
dvc init
dvc add data/raw                   # writes data/raw.dvc and git-ignores data/raw
git add data/raw.dvc data/.gitignore
git commit -m "Track raw data v1"
dvc remote add -d storage s3://my-bucket/dvc
dvc push                           # upload the data itself

# on another machine
git pull && dvc pull

DVC also defines pipelines in dvc.yaml (stages with inputs, outputs and parameters); dvc repro reruns only the stages whose inputs changed. In November 2025 lakeFS acquired the DVC open-source project; DVC stays open source and aimed at individual projects, while lakeFS targets data-lake scale (lakeFS announcement).

The Hugging Face Hub stores every model and dataset as a Git repository. Large files are still referenced by LFS-style pointer files, but the Hub has moved their storage from Git LFS to Xet, which deduplicates at the chunk level so only changed bytes are uploaded (Hub storage docs). For large repositories, use the huggingface_hub library rather than a plain git clone.

The Part 3 lesson applies here too. A branch like main on a model repository can move, while a commit hash cannot. Pin it: from_pretrained("org/model", revision="<commit-sha>").

The commit hash is your experiment ID

To reproduce a training run you need the code (commit hash), whether uncommitted edits were present, the data version, the config, the environment lockfile, and the seeds. Record them all with every run.

import subprocess

def git_state() -> dict:
    sha = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
    dirty = subprocess.check_output(["git", "status", "--porcelain"], text=True) != ""
    return {"git_sha": sha, "git_dirty": dirty}

MLflow and Weights & Biases record the commit automatically when a run starts inside a repository. A good rule: never launch a serious run from a dirty tree. If you must, save the uncommitted diff beside the run with git diff HEAD > runs/<id>/uncommitted.patch.

Experiments as branches

Branches are free, so exp/lora-r16 for an experiment costs nothing. Keep results in the tracker, not in branch names, and delete dead branches. Tag the commits behind results you report or ship. Compare two runs’ configurations with git diff <sha1> <sha2> -- configs/.

Notebooks

An .ipynb file is JSON holding code, outputs, execution counts and embedded images. Diffs are noisy, merges conflict constantly, files balloon, and outputs can leak data or keys.

• Strip outputs on commit: nbstripout --install sets up a Git filter so outputs never enter history.
• Pair with scripts: Jupytext keeps a .py twin of each notebook that diffs and reviews like normal code.
• Diff and merge properly: nbdime config-git --enable makes git diff notebook-aware.
• Move stable logic into modules. Notebooks should import and call it, not contain it.

Secrets

A secret is compromised the moment it is pushed. Deleting it in the next commit does not help: it remains in history, and automated scanners watch public repositories continuously.

1. Revoke and rotate the key first, at the provider. This is the only step that actually protects you.
2. Then remove it from history with git filter-repo, the tool the Git project recommends, and force-push.
3. Check the provider’s logs for misuse.

Prevention is cheap. Keep .env in .gitignore and commit a .env.example with dummy values. Add a secret scanner such as gitleaks to your pre-commit hooks (Part 11), and enable your host’s push protection.

Check yourself

• Why does deleting a 3 GB checkpoint in a new commit not shrink the repository?
• What must you record with a training run so someone can reproduce it exactly?
• You pushed an API key ten minutes ago. What is step one, and why does it come before cleaning history?