A script that saves its output next to its own source file writes into whichever checkout runs it. Run it from a linked worktree, remove the worktree, and the output is gone, with no warning from git.
Path(__file__).parent / "data", points inside the linked worktree the script was started from. If that folder is listed in .gitignore, git worktree remove does not count it as uncommitted work, so it deletes the folder and exits 0 without a warning. The fix is to build runtime paths from the main checkout, which is the parent of git rev-parse --git-common-dir, with an environment variable to override it. Templates, read-only committed data and files you commit on purpose can stay where they are.Table of Contents
The symptom: output files missing after a worktree was removed
You run a fetch job, an export or a scraper from a git worktree. It writes captures, a cache, a SQLite file or a set of reports into a data/ or output/ folder in the project, and that folder is in .gitignore because it holds generated or sensitive files. The work merges, the worktree is cleaned up, and a day later the files are gone. Anything that recorded their paths, such as a database row, a manifest or a log, still points at a folder that no longer exists.
It is easy to misdiagnose. Before you recreate anything, check whether the data is really lost or whether you are looking in a different checkout from the one that wrote it (real loss or a stale read walks through that check). If the files were written inside a worktree that has since been removed, they are really gone.
This happens more often now that parallel work runs in worktrees by default. Giving each coding agent or task its own git worktree stops them from overwriting each other’s files. It also means a script is more likely to run from a short-lived directory than from the main checkout.
Reproducing it in a throwaway repository
A script that does what many project scripts do:
# tools/fetch.py
from pathlib import Path
DATA_DIR = Path(__file__).resolve().parent.parent / "data"
DATA_DIR.mkdir(exist_ok=True)
(DATA_DIR / "captures.jsonl").write_text('{"id": 1}\n')
print("wrote", DATA_DIR / "captures.jsonl")
Code language: Python (python)
With data/ in .gitignore, create a worktree, run the script there, then remove the worktree from the main checkout:
$ git worktree add ../app-task -b task
$ cd ../app-task && python tools/fetch.py
wrote .../app-task/data/captures.jsonl
$ git status --short
# clean: data/ is ignored
$ cd ../app && git worktree remove ../app-task
$ echo $?
0
$ ls ../app-task data
ls: cannot access '../app-task': No such file or directory
ls: cannot access 'data': No such file or directoryCode language: PHP (php)
The captures existed only in the worktree, and the removal deleted them along with it. The main checkout never had a copy.
git worktree remove protects your uncommitted work, but ignored files do not count as work.
Why git does not warn you
The git worktree documentation says only clean worktrees can be removed, meaning no untracked files and no changes to tracked files; --force overrides that. “Untracked” excludes ignored paths. A file matched by .gitignore is outside git’s view entirely, so it neither blocks the removal nor triggers a warning. The same is true when a worktree directory is deleted by hand, or by a cleanup tool that runs git worktree prune afterwards.
That is reasonable from git’s side. Ignored files are usually build output that can be regenerated. Captured responses, downloaded archives and run logs cannot.
Which paths risk accidental data loss
Any runtime path whose root is the checkout the code runs from, rather than a fixed location, is at risk. The common forms:
__file__anchors:Path(__file__).parent / "data",os.path.join(os.path.dirname(__file__), "output"),parents[1] / "exports".- Named roots built from
__file__:BASE_DIR = Path(__file__).resolve().parent.parentin one module, thenBASE_DIR / "media"orBASE_DIR / "backups"somewhere else. - Environment defaults:
os.getenv("DATA_DIR", BASE_DIR / "data")looks configurable, but whenever the variable is unset it falls back to the worktree. - Current-directory paths:
"./output"orPath.cwd() / "cache"land wherever the shell happened to be.
What decides the risk is the first folder under the root. BASE_DIR / "data" / "captures" is runtime data. BASE_DIR / "scripts" / "backup" is a code folder that happens to contain the word “backup”.
The fix: anchor runtime data to the main checkout
Every worktree shares one repository directory, and git reports it through git rev-parse --git-common-dir. In the main checkout that directory is <main>/.git; in a linked worktree it is the same <main>/.git, so its parent is the main checkout from either place. One detail matters: in the main checkout git answers with the relative path .git, so resolve it against the folder you ran git in.
# tools/data_root.py
import os
import subprocess
from pathlib import Path
CODE_ROOT = Path(__file__).resolve().parent.parent
def primary_checkout() -> Path:
"""The main checkout, even when this code runs from a linked worktree."""
out = subprocess.run(
["git", "rev-parse", "--git-common-dir"],
cwd=CODE_ROOT, capture_output=True, text=True, check=True,
).stdout.strip()
common = Path(out)
if not common.is_absolute(): # the main checkout answers ".git"
common = CODE_ROOT / common
return common.resolve().parent
def data_dir(env_var: str = "APP_DATA_DIR") -> Path:
raw = os.environ.get(env_var)
if raw:
root = Path(raw)
return root if root.is_absolute() else primary_checkout() / root
return primary_checkout() / "data"
Code language: Python (python)
The script then asks for data_dir() instead of building its own path. Run from the same worktree, it now writes to the main checkout, and the data survives the removal:
$ cd ../app-task3 && python tools/fetch.py
wrote .../app/data/captures.jsonl
$ APP_DATA_DIR=shared python tools/fetch.py
wrote .../app/shared/captures.jsonl
$ cd ../app && git worktree remove ../app-task3 && ls data shared
data:
captures.jsonl
shared:
captures.jsonl
Path(__file__).parent / data resolves inside the worktree the script was run from.
data/ is ignored, so it is not untracked work: deleted, exit 0, no warning.
Parent of –git-common-dir. Every worktree writes there; removal leaves the data.
An environment variable can move the data root anywhere; a relative value is resolved against the main checkout, never the current directory.
Three rules keep the fix from reopening the hole:
- Resolve a relative override against the main checkout. If
APP_DATA_DIR=datawere used as given, it would be relative to the current directory, which is the worktree again. - Fail rather than guess when git is unavailable. Falling back to “the folder this file is in” puts the data straight back into the worktree. Either follow the worktree’s
.gitfile (it contains agitdir:line pointing into the main repository), or stop with an error that tells the user to set an absolute override. - Keep one resolver. When every script calls the same function, a reviewer can check one place instead of every path in the codebase.
In a shell script, git rev-parse --path-format=absolute --git-common-dir (git 2.31 and later) returns an absolute path directly, so "$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")" is the main checkout.
The fix is not a safer path. It is one function that every runtime path goes through.
Paths that should stay inside the checkout
Not every __file__-relative path is a bug, and “fixing” the wrong ones makes things worse:
- Templates and code assets (
templates/,static/, a JSON schema shipped with the code) belong to the checkout. A worktree on a branch that changes a template must read its own copy. - Read-only committed data, such as a lookup table in the repository, is versioned with the code. Reading it from the worktree is correct.
- Tracked working sets that are meant to travel with the branch and get committed are safe, because the commit is the copy. Only write there if the change really is committed before the worktree goes.
- Tests that build a path to compare against, without writing to it, do not move any data.
For each case, ask whether the folder is ignored and written at run time. If it is, it belongs under the main checkout or an external data root. If it is committed, or only read, it can stay.
Finding the risky paths in an existing codebase
A text search finds most candidates in a few seconds:
git grep -nE "__file__|BASE_DIR|PROJECT_ROOT|getcwd|Path\.cwd" -- "*.py" \
| grep -iE "data|output|runtime|cache|export|download|backup|logs|store"Code language: Bash (bash)
Expect noise. Review each hit by asking three questions: is the folder ignored, is it written at run time, and is it the first folder under the root? If you want a check that blocks new cases in review or in a pre-push hook, a plain search is too noisy to enforce. It needs to:
- follow names such as
BASE_DIRandROOTback to__file__or the current directory; - treat environment-variable defaults as paths;
- skip the resolver function itself;
- flag only a data word sitting directly under the root, so
ROOT / "scripts" / "backup"passes; - accept a waiver comment on the line that states a reason, for the committed and read-only cases above.
A check without a waiver blocks legitimate code; a waiver without a reason hides the next bug.
Turning on a blocking check without breaking everyone
When the check blocks pushes or fails CI across several repositories, the order of the rollout matters more than the check itself:
- Fix or waive every existing hit first, starting with any repository whose CI pulls the check from its shared source. If the check lands first, those pipelines turn red on code nobody touched.
- Then merge the check, and confirm each repository scans clean on its default branch.
- Then install the local hook on developer machines.
Two gaps remain after that. A pre-push hook never runs in a repository with no remote, because nothing is ever pushed, so scan those repositories by hand. And a hook that scans the whole repository will reject a feature branch created before the fix, because the branch still contains the old lines. Merging the default branch into it clears that. Pull-request CI is usually unaffected, because on GitHub the pull_request event checks out the branch merged with the current base.
The short version
A data path built from __file__ or the current directory lands inside whichever worktree runs the code. If that folder is ignored, git worktree remove deletes it with exit 0 and no warning. Resolve runtime data from the main checkout, the parent of git rev-parse --git-common-dir, with an environment override that is never resolved against the current directory, and route every write through that one function. Leave templates, read-only committed data and committed working sets where they are. If you enforce it with a blocking check, fix every existing hit before the check lands and install hooks last.
Frequently Asked Questions
Can I recover files deleted by git worktree remove?
Not through git. Ignored files were never added to the repository, so no commit, stash or reflog entry holds them. The removal also bypasses the operating system’s recycle bin. Your options are a filesystem backup, a snapshot of the disk, or an undelete tool used before the space is reused. Rebuild from the original source if none of those exists.
Is the main checkout a safe place for runtime data?
It is safe from worktree removal but not from everything. git clean -fdx in the main checkout deletes ignored files too, and a fresh clone starts without them. For data that matters, point the environment override at a folder outside the repository and back that folder up. The main-checkout default protects you from the common accident; it is not a backup.