Why Your Pre-Push Hook Gets Slower Every Week, and What Fixes It

Date:

Every feature arrives with its tests. Nobody times the hook. Then one day git push takes 25 minutes for a one-line change, and the suite is still running one test at a time on a 16-core machine.

The short answer: a pre-push hook gets slow because the test suite grows with the code while the hook keeps running it the way it did on day one, one test after another. Fix it in order. First profile the suite and remove the few tests that waste time waiting. Then run tests in parallel: one Python suite dropped from 726 to 443 seconds with that switch alone. Then make sure parallel workers share the queue, so a worker that finishes early takes tests from a busy one (work stealing) instead of sitting idle. Last, stop the hook from running tests that cannot tell you anything new: a push that only edits docs, or files that already passed the whole suite. Each skip must be narrow, fail toward running, and leave a log line.

Why a pre-push hook gets slower every week

Git runs the pre-push hook before a push leaves your machine, and stops the push if the hook fails. It is a free, local place to run the tests, so it often holds one line: pytest, or the project’s own test runner.

Nobody runs the hook on purpose, so nobody times it. Meanwhile every feature adds tests. One Python suite went like this:

DateTestsOne full run
September1,493serial
early October1,600726 s serial (12 minutes)
mid November2,725383 s, already parallel

A PHP suite on the same machine reached about 700 tests and 1,473 seconds, about 25 minutes, while still running one test at a time. A full run had already taken about 19 minutes on a cloud container weeks earlier; the choice then was to keep it serial.

That is the trap. Test runners run one test at a time unless told otherwise, the default never announces itself, and a suite that took two minutes when the project began takes twenty a few hundred tests later. By the time it hurts, the fix is no longer one change but several, and they work best in this order.

1. Profile the slow test suite before adding cores

Find out where the time goes first. With pytest, --durations=30 lists the slowest tests; with any runner, timestamps around each step show whether the time is in the tests at all.

The 12-minute Python suite was not 1,600 equally slow tests. Two tests each walked and re-parsed the whole source tree, about 217 seconds between them. One connected to a database with no timeout and hung 130 seconds before failing. Later, four tests waited about 30 seconds each because a killed background job’s child sleep held their output pipe open, and two more slept through real delays in the code they tested (a 30-second sampling interval, six 10-second retry backoffs). Each fix was in the test or its setup, and every test still runs.

A test that sleeps for real is timing your patience, not your code.

2. Stop running tests one at a time

On a multi-core machine, a serial suite uses one core. Running tests in parallel is the biggest single lever:

  • For pytest, the pytest-xdist plugin adds -n auto, one worker process per core. The 1,600-test suite went from 726 to 443 seconds with that flag alone, and to about 250 to 300 seconds with the profiling fixes from step 1.
  • For a runner without a parallel mode, split the tests into groups that share nothing and run each group as its own process. The PHP suite ran every test against three database engines, each with its own server, so the three engine groups became three processes, then the engine-free tests ran alone. A full run went from 1,473 to 657 seconds, and every test still ran on every engine.

Two problems surface the first time a suite runs in parallel:

  • Order dependence. One PHP test had passed for months only because an earlier test, in serial order, cleared a row it relied on. Run apart, it failed; it now starts from an empty database. Serial order had been hiding a broken test.
  • Pipes on Windows. Send each worker’s output to a file, not a pipe. A child process can inherit the pipe’s handle and keep it open, and the parent then waits for a close that never comes; one such wait lasted three hours.

3. Let an idle worker take tests from a busy one

Parallel is not the end of it. Look at how long each worker actually worked. In the PHP suite’s split, each engine group was fixed in advance:

A bar chart of how long each of three parallel workers took on its fixed share of one test suite, on a 16-core machine. The MySQL group took 524 seconds, the PostgreSQL group 445 and the MariaDB group 384. The MariaDB worker, flagged, then sat idle for 140 seconds while the MySQL worker still had tests queued, because a fixed split gives an idle worker nothing to take. A note says the run only ends when the slowest worker finishes.

The run ends when the slowest worker ends. The MariaDB worker finished its share after 384 seconds and then sat idle for 140 seconds while the MySQL worker still had tests waiting. Nothing in a fixed split lets it help. Here the MySQL tests needed their own server, but about 120 seconds of tests needed no database at all, and they waited until every group was done.

The fix is to share the queue. The worker that runs out should take tests still waiting on another worker’s queue. That is work stealing, and pytest-xdist offers it as --dist worksteal. Tests start spread evenly across the workers. When a worker is down to its last few, it takes a share of the tests queued on a busy worker. It suits a suite whose tests vary widely in duration, which is most suites that grew for a while.

How much it saves depends on the suite, so time it. On the Python suite, timed twice each on the same code and the same 16-core machine, the default scheduler took 197 and 214 seconds and work stealing took 185 and 204: about 5% faster. After step 1 that suite had few long tests left, so there was little imbalance to recover; the gain grows with how unequal the tests are. One test failed in both work-stealing runs and passed when run alone. It gave another process a fixed 3 seconds to act. With every worker busy until the last test, the machine stays loaded to the end, and a fixed timing window is exactly what a loaded machine breaks. Fix the test, not the scheduler.

A suite split by hand can get the same effect without a plugin. Cut the work into many small units, test files for example, put them in one shared list, and let each worker take the next unit when it finishes one. Then no worker is idle while another still has a queue.

4. Let one push run its suite at a time

Several pushes can start at once on one machine: a few terminals, a few branches, a few AI coding sessions. Each hook starts the whole suite, and now each suite also wants every core. Three at once made each slower than running them in turn, and the burst of new processes kept setting off the antivirus scanner.

A machine-wide lock around the suite step fixes it:

lock=/tmp/suite.lock     # one per machine, shared by every repository
until mkdir "$lock" 2>/dev/null; do sleep 30; done
trap 'rmdir "$lock"' EXIT
run_the_whole_suite
Code language: Bash (bash)

mkdir succeeds for exactly one caller, so it is a lock that needs no tools. Take it only around the suite, so a push that skips the suite never waits. Break a lock whose owner is gone or which is older than any real run, or one crashed hook stops every push on the machine.

5. Skip a push that only adds or edits documentation

The fastest suite is the one that does not run. Git hands the hook one line per pushed ref on standard input, in the form the githooks documentation gives: <local ref> <local sha> <remote ref> <remote sha>. Comparing the two commits says which files the push changes:

while read -r local_ref local_sha remote_ref remote_sha; do
  git diff --name-status -M "$remote_sha" "$local_sha"
done
Code language: Bash (bash)

If every line is an added (A) or modified (M) file under docs/ or ending in .md, nothing the tests execute has changed. Keep the rule narrow:

  • A deleted or renamed doc still runs the suite. A test may open that file by name: a link checker, a test that reads the changelog.
  • A new branch has no remote commit to compare with (git sends a sha of zeros). Compare with the main branch instead, or run the suite.
  • If some tests read the docs, run just those. A grep at push time finds every test file that mentions docs/ or .md. In the Python suite they took 22 seconds, instead of about 8.5 minutes for the whole suite.

6. Never run the same files twice

The docs rule compares with what the remote already has, so it cannot see a run that repeats a pass. You ran the suite by hand, then push a new branch. A push was refused after the suite passed, and you push again. The code is identical to code that passed minutes ago.

Git already names “these exact files”: the tree hash. git rev-parse HEAD^{tree} identifies the content of a commit regardless of history, so a rebase that lands on the same files gets the same hash. Record a hash when the whole suite passes on it; when a later push arrives, walk back through recent commits, and if one has a recorded pass and only docs changed since, skip.

store="$(git rev-parse --path-format=absolute --git-common-dir)/suite-passed"
tree="$(git rev-parse 'HEAD^{tree}')"
# record only a clean, whole-suite pass
[ -z "$(git status --porcelain --untracked-files=all)" ] &&
  mkdir -p "$store" && printf 'python=%s\n' "$(python -V 2>&1)" > "$store/$tree"
Code language: Bash (bash)

A record is a promise, so write it only when the pass was real: the whole suite ran (no file list, no -k filter), the checkout was clean at the start and the end, and the interpreter version matches. The git common directory is shared by every linked worktree, so a pass recorded in one saves the run in another.

A recorded pass is a promise that these exact files passed. Write it only when that is true.

In the PHP suite, a code push ran the whole suite (810 seconds) and recorded its tree, and the docs push after it took 7 seconds. In a mid-size Python suite, the same pair took 86 seconds and then 1.

7. Fail toward running, and log every skip

Any rule above can be wrong in a way nobody foresaw, so the default is the safe one: a push line the hook cannot parse, an error from git or an empty change list runs the suite. Keep an escape hatch, such as FULL=1, that forces a run.

Does this push need the whole suite?Three questions, asked in order. Any doubt at all means the suite runs.
1. Read the push

One line per ref: local sha, remote sha. A line it cannot read runs the suite.

2. Only docs added or edited?

Yes: skip. A deleted or renamed doc still runs: a test may read that file.

3. Passed already?

Same files, clean tree, same interpreter, only docs since: skip. Else run, then record.

keyed by tree hash

Every skip appends one line, with its reason, to a log beside the records, so the time it saved can be counted rather than guessed.

Print the reason for each skip, and append it with a timestamp to a log beside the records. After a week, count the lines and multiply by the suite’s run time: the saving, measured instead of guessed. Then prove each rule as you would any test: remove the rule and watch a test fail.

One trap waits once the hook’s own tests build throwaway repositories. When you push from a linked worktree, git exports GIT_DIR into the hook, and every git command the hook starts inherits it, whatever folder it runs in. A test’s git init and git config then write to the real repository: here, core.bare = true and a fake user in the shared config, which broke every checkout at once. Clear the variables git lists for this at the start of the test runner:

unset $(git rev-parse --local-env-vars)Code language: Bash (bash)

What belongs in a hook and what is left for CI is covered in git hooks vs GitHub Actions. If several people or agents push from one machine, start with one git worktree per task.

A checklist for a faster pre-push hook

  1. Have you timed the suite lately? It grows with every feature.
  2. Does it still run serially? Use every core.
  3. Do workers finish together? If one idles while another has a queue, share the queue.
  4. Can two pushes on one machine run the suite at once? Lock the suite step only.
  5. Does a docs-only push skip, and a tree that already passed? Narrowly, and logged.
  6. Does anything unreadable run the suite?

The short version

  • A test suite grows with the code; the hook keeps running it serially until someone notices.
  • Profile first, then run in parallel (726 to 443 seconds from the flag alone), then let idle workers steal queued tests from busy ones.
  • Then skip only what cannot fail: docs-only pushes and files whose tree hash already passed. Log every skip.

Frequently Asked Questions

Is it safe to skip tests when CI runs them anyway?

If CI runs the same suite on every pull request, the pre-push skip only moves the proof, and the risk is small. If the hook is your only gate, the skip has to stand on its own. That is why it is limited to two cases where the outcome cannot differ: no executed file changed, or these exact files already passed.

What if the tests depend on something outside the repository?

The tree hash covers your files, not the machine. The record above also names the interpreter version, so a Python or PHP upgrade runs the suite. A database version or an operating system update is not in the key. Add whatever your suite is sensitive to into the record, or force a run with the escape hatch after changing the machine.

Share post:

Subscribe

Popular

More like this
Related

Can AI CLI Conversation History Leak Answers Into a Blind Test?

You use an AI command-line tool to grade or answer hundreds of questions, one fresh run each. Then you find out it keeps every conversation on disk, answer keys included. Is run 200 quietly reading run 199?

The Code Snippets 403 That Comes From Your Host’s Firewall, Not WordPress

You change a few lines in a snippet, press Save, and get "Could not update snippet. Request failed with status code 403". Press Save on the unchanged snippet and it works. Your login is fine. Something between you and WordPress is reading the code itself.

Why a Flaky Test Passes Alone and Fails When the Machine Is Busy

Run the test by itself and it passes, every time. Run the whole suite in parallel and it fails, but only some days, and its failure looks exactly like a real bug. In our case the code was fine. The test was waiting on a clock.

Why Multisite Subsite Registration Sends Visitors to the Main Site

You put a "create an account" link on one site of a WordPress multisite. Visitors who click it land on the main site, which either says "Registration has been disabled" or signs them up there instead.