Run the test by itself and it passes, every time. Run the whole suite in parallel and it fails, but only some days, and its failure looks exactly like a real bug. In our case the code was fine. The test was waiting on a clock.
sleep, a timer or a fixed timeout assumes the program under test will reach a point within so many seconds. On an idle machine it always does. When every core is busy, start-up slows down, the program arrives late, and the test checks before anything has happened. To fix it, find the clock and replace it with a signal from the program itself: a line it prints, a file it writes, a port it opens. Keep a long deadline only to catch a real hang. Then prove the new test still fails when the behavior is broken, because a flaky failure and a real bug can print exactly the same thing.Table of Contents
The symptom: green alone, red in the parallel run
The test in question checks a deploy script that must not run while another copy is running. The script takes a lock by creating a folder; mkdir either creates it or fails, so only one copy can hold it. If the folder already exists, the script waits:
# deploy.sh (excerpt): wait while another run holds the lock
waited=0
while ! mkdir "$LOCK" 2>/dev/null; do
[ "$waited" = 0 ] && echo "waiting for the lock"
sleep 5
waited=$((waited + 5))
done
[ "$waited" = 0 ] || echo "lock taken after ${waited}s"
Code language: Bash (bash)
The test plays the other copy. It creates the lock folder itself, starts a timer to delete the folder three seconds later, runs the script, and checks that the script waited:
hold_lock(lock) # pretend another run has it
threading.Timer(3.0, lambda: shutil.rmtree(lock)).start()
r = subprocess.run(["sh", "deploy.sh"], capture_output=True, text=True, timeout=120)
assert "waiting for the lock" in r.stdout
assert "lock taken after" in r.stdout
Code language: Python (python)
On its own it passed every time, in about 11 seconds. Then the suite of about 2,800 tests started running in parallel with pytest-xdist, which splits the tests across one worker process per core. Here is what the same test did, on the same code and the same 16-core machine, on one day:
| How it ran | Result |
|---|---|
| Alone, idle machine | passed |
Whole suite in parallel, --dist load, 2 runs | passed both |
Whole suite in parallel, --dist worksteal, 2 runs | failed both |
Both failures printed the same thing:
AssertionError: assert 'waiting for the lock' in 'starting deploy\n'Code language: JavaScript (javascript)
The script had run, printed its first line, and finished without ever waiting. Read cold, that says the script’s lock is broken. It wasn’t.
A flaky failure and a real bug can print exactly the same line.
The rest of this article goes from that failure to a test that can’t flake. Each step is settled by a test run rather than a guess.
Load every core, run the test alone, until it goes red every time.
A sleep, a timer or a fixed timeout that assumes the program is ready in time.
A line it prints, a file it writes, a port it opens. Keep a long deadline for a hang.
Remove the behavior: the test must go red. Restore it: green under the same load.
A clock in a test is a guess about how fast the machine is. A busy machine is where the guess is wrong.
Step 1: make it fail on demand
A test that fails one run in five can’t be debugged, because a pass proves nothing. So the first job is a way to make it fail every time. If the suspicion is load, create load: start more busy processes than there are cores, then run the one test by itself.
# burn.py: keep N processes spinning for S seconds
import multiprocessing as mp, sys, time
def spin(seconds):
end = time.time() + seconds
while time.time() < end:
pass
if __name__ == "__main__":
n, seconds = int(sys.argv[1]), float(sys.argv[2])
procs = [mp.Process(target=spin, args=(seconds,)) for _ in range(n)]
for p in procs: p.start()
for p in procs: p.join()
Code language: Python (python)
With 32 spinning processes on the 16-core machine, the test alone failed 2 runs out of 2, with the same output as the suite failures. When the load stopped, the next 2 runs passed. That settles the cause class: it’s load, not test order, not another test’s leftovers, not the code. A test that only fails next to other tests, whatever the load, would point to shared state instead, which is a different hunt.
Step 2: find the clock in the test
With a reliable failure in hand, read the test for anything that measures time. Here it is the threading.Timer(3.0, ...) line. Python’s Timer runs a function after a delay in its own thread, and it starts counting the moment .start() is called, which is before the script has even been launched.
So the test quietly assumed that within three seconds the script would have started, read its input, done its setup and reached its mkdir. On an idle machine that takes well under a second. On a machine where every core is already busy, starting a shell and the commands it calls can take longer than three seconds, and here it evidently did. When that happens, the timer deletes the lock first, the script finds the folder gone, takes the lock at once, and never prints “waiting”. The test then fails on a script that worked exactly as written.
The timer didn’t measure the script. It measured how busy the machine was.
Why did one scheduler expose it and not the other? We didn’t prove that part, so treat this as the likely reason. In pytest-xdist’s worksteal mode, a worker that runs out of tests takes some from a worker that still has a queue, so every core stays busy until the very end of the run. That’s its purpose, and on this suite it was a few seconds faster than load mode. It also means a test that lands late in the run meets a machine that’s fully loaded. The scheduler didn’t cause the bug. It removed the quiet moments the bug had been hiding in.
The same shape shows up as time.sleep(2) before an assertion, a fixed 5-second wait for a server to come up, or a short timeout= on a call that’s normally quick. Each one is a guess about the machine’s speed, and each one passes for months on a developer’s laptop.
Step 3: wait for the program’s own signal
The script already says when it has reached the interesting point: it prints “waiting for the lock”, and only after mkdir has failed. So the test can wait for that line instead of a clock. That means reading the script’s output while it runs, rather than after it exits:
def run_script(on_line, timeout=120):
"""Run deploy.sh; call on_line(line) for each line as the script prints it."""
proc = subprocess.Popen(["sh", "deploy.sh"], stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True)
out = []
def pump():
for line in proc.stdout:
out.append(line)
on_line(line)
reader = threading.Thread(target=pump, daemon=True)
reader.start()
try:
rc = proc.wait(timeout=timeout)
except subprocess.TimeoutExpired:
proc.kill()
raise
reader.join(timeout=10)
return rc, "".join(out)
def release_when_waiting(line):
if "waiting for the lock" in line:
shutil.rmtree(lock) # the other run finishes: free the lock
hold_lock(lock)
rc, out = run_script(release_when_waiting)
assert rc == 0
assert "waiting for the lock" in out and "lock taken after" in out
Code language: Python (python)
Now the lock is released only after the script has proved it met a held lock. However slow the machine, the order of events is fixed. The reader thread matters: reading a child’s output in the same thread that waits for it can deadlock once the pipe fills, which is why the subprocess docs steer you to communicate() or to reading the pipe as it fills.
The 120-second deadline is still there, but its job has changed. It no longer decides when things happen. It only stops a script that hangs, or one that waits without saying so, from blocking the suite forever. A deadline that generous never fires on a working program, however busy the machine.
If the program prints nothing useful, wait on something else it does that you can observe: a file appearing, a port accepting connections, a row in a table. Check it in a loop every fraction of a second, with the same long deadline. That approach is slower than reading a printed line, but it’s still keyed to the program’s progress instead of the wall clock. Martin Fowler’s Eradicating Non-Determinism in Tests makes the same case for asynchronous tests: never wait a fixed time for something you can poll for or be told about.
Step 4: break the code and require red
A changed test needs proof that it can still fail. A rewrite that waits for the “waiting” line could, through some mistake, pass whether or not the script waits, and a test that can’t fail is worse than a flaky one: it looks reassuring. The check is the one described in why a test that never fails might be testing nothing: break the behavior the test is named after, and watch for red.
Here that meant replacing the script’s whole wait loop with a bare mkdir, so it never waits:
mkdir "$LOCK" 2>/dev/null # broken on purpose: no wait for a held lockCode language: Bash (bash)
The new test failed against that broken script, with this output:
AssertionError: assert 'waiting for the lock' in 'starting deploy\n'Code language: JavaScript (javascript)
That’s character for character the flaky failure from the parallel runs. Before the fix, nobody could tell a slow machine from a broken lock by reading the output. After it, that output can only mean the lock is broken. With the script restored, the test passed again.
After the fix, a red test can only mean one thing.
Step 5: rerun under the load that broke it
The last check repeats the conditions that produced the failure, not just the happy path:
| Check | Before | After |
|---|---|---|
| The test alone, 32 busy processes | failed 2 of 2 | passed 4 of 4 (33 to 59 s each) |
Whole suite in parallel, --dist worksteal | failed 2 of 2 | passed 4 of 4 (2,797 tests each) |
| Script’s wait loop removed | (not run) | fails, as it should |
The slower times under load show the machine really was saturated during those passes. A pass on an idle machine would only have repeated the original false comfort.
A checklist for clocks in tests
When a test passes alone and fails on a busy machine, or only in CI:
- Make it fail on demand. Load every core (more busy processes than cores) and run the test alone. If it fails every time, the cause is timing; if it doesn’t, look at shared state between tests.
- Search the test for time.
sleep,Timer,wait(…)with a short value,timeout=under a few seconds, “retry 3 times”. Ask what each one assumes will have happened by then. - Replace the clock with a signal. Wait for a printed line, a file, a port or a database row. Read output while the program runs, on a separate thread.
- Keep one generous deadline, sized to catch a hang, never to pace the test.
- Break the behavior and require red, then restore it and require green.
- Rerun under the load that broke it, several times. One pass proves little; four in a row under load is evidence.
This is one cause among several of code that works on one machine and fails on another. Machine speed is the one you can’t see in the code.
The short version
A test that passes alone and fails on a busy machine is usually timing the machine. Reproduce it by loading every core, find the sleep or timer, and replace it with a signal the program gives once it has reached the point the test cares about. Keep a long deadline only for hangs. Then break the code on purpose to prove the test still goes red, because the old flaky failure and a real bug printed the same line.
Frequently Asked Questions
Why not just raise the timer from 3 seconds to 30?
Because it moves the guess without removing it, and it makes every run slower. A 30-second wait costs 30 seconds on every run, even on an idle machine, and a busy enough machine (a shared CI runner, a laptop on battery) can still miss it. A signal costs nothing extra when the program is fast and waits as long as needed when it’s slow.
Is retrying a flaky test automatically a reasonable fix?
Retrying hides the symptom and keeps the cause. Tools that rerun failed tests are useful as a short stopgap while you find the clock, but a test that needs retries to pass will one day fail on a real bug and get retried into green. If you use retries, log every one, so a test that starts needing them stands out.
What if the program gives no signal I can wait for?
Add one. A single line printed when the program reaches an important state (“listening on port 8080”, “waiting for the lock”) costs nothing and helps people reading logs as much as it helps tests. If you can’t change the program, poll something it does change (a file, a port, a table) with a long deadline. That still ties the test to the program’s progress, not to the clock.