Why a Flaky Test Passes Alone and Fails When the Machine Is Busy

Date:

Run the test by itself and it passes, every time. Run the whole suite in parallel and it fails, but only some days, and its failure looks exactly like a real bug. In our case the code was fine. The test was waiting on a clock.

The short answer: a flaky test that passes alone and fails only when the machine is busy is very often timing something it does not control. Somewhere in it a sleep, a timer or a fixed timeout assumes the program under test will reach a point within so many seconds. On an idle machine it always does. When every core is busy, start-up slows down, the program arrives late, and the test checks before anything has happened. To fix it, find the clock and replace it with a signal from the program itself: a line it prints, a file it writes, a port it opens. Keep a long deadline only to catch a real hang. Then prove the new test still fails when the behavior is broken, because a flaky failure and a real bug can print exactly the same thing.

The symptom: green alone, red in the parallel run

The test in question checks a deploy script that must not run while another copy is running. The script takes a lock by creating a folder; mkdir either creates it or fails, so only one copy can hold it. If the folder already exists, the script waits:

# deploy.sh (excerpt): wait while another run holds the lock
waited=0
while ! mkdir "$LOCK" 2>/dev/null; do
    [ "$waited" = 0 ] && echo "waiting for the lock"
    sleep 5
    waited=$((waited + 5))
done
[ "$waited" = 0 ] || echo "lock taken after ${waited}s"
Code language: Bash (bash)

The test plays the other copy. It creates the lock folder itself, starts a timer to delete the folder three seconds later, runs the script, and checks that the script waited:

hold_lock(lock)                                     # pretend another run has it
threading.Timer(3.0, lambda: shutil.rmtree(lock)).start()
r = subprocess.run(["sh", "deploy.sh"], capture_output=True, text=True, timeout=120)
assert "waiting for the lock" in r.stdout
assert "lock taken after" in r.stdout
Code language: Python (python)

On its own it passed every time, in about 11 seconds. Then the suite of about 2,800 tests started running in parallel with pytest-xdist, which splits the tests across one worker process per core. Here is what the same test did, on the same code and the same 16-core machine, on one day:

How it ranResult
Alone, idle machinepassed
Whole suite in parallel, --dist load, 2 runspassed both
Whole suite in parallel, --dist worksteal, 2 runsfailed both

Both failures printed the same thing:

AssertionError: assert 'waiting for the lock' in 'starting deploy\n'Code language: JavaScript (javascript)

The script had run, printed its first line, and finished without ever waiting. Read cold, that says the script’s lock is broken. It wasn’t.

A flaky failure and a real bug can print exactly the same line.

The rest of this article goes from that failure to a test that can’t flake. Each step is settled by a test run rather than a guess.

From a flaky test to a deterministic oneEach step is settled by a test run rather than a guess.
1. Fail on demand

Load every core, run the test alone, until it goes red every time.

2. Find the clock

A sleep, a timer or a fixed timeout that assumes the program is ready in time.

3. Wait for the signal

A line it prints, a file it writes, a port it opens. Keep a long deadline for a hang.

the fix
4. Break it, then load it

Remove the behavior: the test must go red. Restore it: green under the same load.

A clock in a test is a guess about how fast the machine is. A busy machine is where the guess is wrong.

Step 1: make it fail on demand

A test that fails one run in five can’t be debugged, because a pass proves nothing. So the first job is a way to make it fail every time. If the suspicion is load, create load: start more busy processes than there are cores, then run the one test by itself.

# burn.py: keep N processes spinning for S seconds
import multiprocessing as mp, sys, time

def spin(seconds):
    end = time.time() + seconds
    while time.time() < end:
        pass

if __name__ == "__main__":
    n, seconds = int(sys.argv[1]), float(sys.argv[2])
    procs = [mp.Process(target=spin, args=(seconds,)) for _ in range(n)]
    for p in procs: p.start()
    for p in procs: p.join()
Code language: Python (python)

With 32 spinning processes on the 16-core machine, the test alone failed 2 runs out of 2, with the same output as the suite failures. When the load stopped, the next 2 runs passed. That settles the cause class: it’s load, not test order, not another test’s leftovers, not the code. A test that only fails next to other tests, whatever the load, would point to shared state instead, which is a different hunt.

Step 2: find the clock in the test

With a reliable failure in hand, read the test for anything that measures time. Here it is the threading.Timer(3.0, ...) line. Python’s Timer runs a function after a delay in its own thread, and it starts counting the moment .start() is called, which is before the script has even been launched.

So the test quietly assumed that within three seconds the script would have started, read its input, done its setup and reached its mkdir. On an idle machine that takes well under a second. On a machine where every core is already busy, starting a shell and the commands it calls can take longer than three seconds, and here it evidently did. When that happens, the timer deletes the lock first, the script finds the folder gone, takes the lock at once, and never prints “waiting”. The test then fails on a script that worked exactly as written.

The timer didn’t measure the script. It measured how busy the machine was.

Why did one scheduler expose it and not the other? We didn’t prove that part, so treat this as the likely reason. In pytest-xdist’s worksteal mode, a worker that runs out of tests takes some from a worker that still has a queue, so every core stays busy until the very end of the run. That’s its purpose, and on this suite it was a few seconds faster than load mode. It also means a test that lands late in the run meets a machine that’s fully loaded. The scheduler didn’t cause the bug. It removed the quiet moments the bug had been hiding in.

The same shape shows up as time.sleep(2) before an assertion, a fixed 5-second wait for a server to come up, or a short timeout= on a call that’s normally quick. Each one is a guess about the machine’s speed, and each one passes for months on a developer’s laptop.

Step 3: wait for the program’s own signal

The script already says when it has reached the interesting point: it prints “waiting for the lock”, and only after mkdir has failed. So the test can wait for that line instead of a clock. That means reading the script’s output while it runs, rather than after it exits:

def run_script(on_line, timeout=120):
    """Run deploy.sh; call on_line(line) for each line as the script prints it."""
    proc = subprocess.Popen(["sh", "deploy.sh"], stdout=subprocess.PIPE,
                            stderr=subprocess.STDOUT, text=True)
    out = []

    def pump():
        for line in proc.stdout:
            out.append(line)
            on_line(line)

    reader = threading.Thread(target=pump, daemon=True)
    reader.start()
    try:
        rc = proc.wait(timeout=timeout)
    except subprocess.TimeoutExpired:
        proc.kill()
        raise
    reader.join(timeout=10)
    return rc, "".join(out)


def release_when_waiting(line):
    if "waiting for the lock" in line:
        shutil.rmtree(lock)          # the other run finishes: free the lock


hold_lock(lock)
rc, out = run_script(release_when_waiting)
assert rc == 0
assert "waiting for the lock" in out and "lock taken after" in out
Code language: Python (python)

Now the lock is released only after the script has proved it met a held lock. However slow the machine, the order of events is fixed. The reader thread matters: reading a child’s output in the same thread that waits for it can deadlock once the pipe fills, which is why the subprocess docs steer you to communicate() or to reading the pipe as it fills.

The 120-second deadline is still there, but its job has changed. It no longer decides when things happen. It only stops a script that hangs, or one that waits without saying so, from blocking the suite forever. A deadline that generous never fires on a working program, however busy the machine.

If the program prints nothing useful, wait on something else it does that you can observe: a file appearing, a port accepting connections, a row in a table. Check it in a loop every fraction of a second, with the same long deadline. That approach is slower than reading a printed line, but it’s still keyed to the program’s progress instead of the wall clock. Martin Fowler’s Eradicating Non-Determinism in Tests makes the same case for asynchronous tests: never wait a fixed time for something you can poll for or be told about.

Step 4: break the code and require red

A changed test needs proof that it can still fail. A rewrite that waits for the “waiting” line could, through some mistake, pass whether or not the script waits, and a test that can’t fail is worse than a flaky one: it looks reassuring. The check is the one described in why a test that never fails might be testing nothing: break the behavior the test is named after, and watch for red.

Here that meant replacing the script’s whole wait loop with a bare mkdir, so it never waits:

mkdir "$LOCK" 2>/dev/null    # broken on purpose: no wait for a held lockCode language: Bash (bash)

The new test failed against that broken script, with this output:

AssertionError: assert 'waiting for the lock' in 'starting deploy\n'Code language: JavaScript (javascript)

That’s character for character the flaky failure from the parallel runs. Before the fix, nobody could tell a slow machine from a broken lock by reading the output. After it, that output can only mean the lock is broken. With the script restored, the test passed again.

After the fix, a red test can only mean one thing.

Step 5: rerun under the load that broke it

The last check repeats the conditions that produced the failure, not just the happy path:

CheckBeforeAfter
The test alone, 32 busy processesfailed 2 of 2passed 4 of 4 (33 to 59 s each)
Whole suite in parallel, --dist workstealfailed 2 of 2passed 4 of 4 (2,797 tests each)
Script’s wait loop removed(not run)fails, as it should

The slower times under load show the machine really was saturated during those passes. A pass on an idle machine would only have repeated the original false comfort.

A checklist for clocks in tests

When a test passes alone and fails on a busy machine, or only in CI:

  1. Make it fail on demand. Load every core (more busy processes than cores) and run the test alone. If it fails every time, the cause is timing; if it doesn’t, look at shared state between tests.
  2. Search the test for time. sleep, Timer, wait(…) with a short value, timeout= under a few seconds, “retry 3 times”. Ask what each one assumes will have happened by then.
  3. Replace the clock with a signal. Wait for a printed line, a file, a port or a database row. Read output while the program runs, on a separate thread.
  4. Keep one generous deadline, sized to catch a hang, never to pace the test.
  5. Break the behavior and require red, then restore it and require green.
  6. Rerun under the load that broke it, several times. One pass proves little; four in a row under load is evidence.

This is one cause among several of code that works on one machine and fails on another. Machine speed is the one you can’t see in the code.

The short version

A test that passes alone and fails on a busy machine is usually timing the machine. Reproduce it by loading every core, find the sleep or timer, and replace it with a signal the program gives once it has reached the point the test cares about. Keep a long deadline only for hangs. Then break the code on purpose to prove the test still goes red, because the old flaky failure and a real bug printed the same line.

Frequently Asked Questions

Why not just raise the timer from 3 seconds to 30?

Because it moves the guess without removing it, and it makes every run slower. A 30-second wait costs 30 seconds on every run, even on an idle machine, and a busy enough machine (a shared CI runner, a laptop on battery) can still miss it. A signal costs nothing extra when the program is fast and waits as long as needed when it’s slow.

Is retrying a flaky test automatically a reasonable fix?

Retrying hides the symptom and keeps the cause. Tools that rerun failed tests are useful as a short stopgap while you find the clock, but a test that needs retries to pass will one day fail on a real bug and get retried into green. If you use retries, log every one, so a test that starts needing them stands out.

What if the program gives no signal I can wait for?

Add one. A single line printed when the program reaches an important state (“listening on port 8080”, “waiting for the lock”) costs nothing and helps people reading logs as much as it helps tests. If you can’t change the program, poll something it does change (a file, a port, a table) with a long deadline. That still ties the test to the program’s progress, not to the clock.

Share post:

Subscribe

Popular

More like this
Related

Can AI CLI Conversation History Leak Answers Into a Blind Test?

You use an AI command-line tool to grade or answer hundreds of questions, one fresh run each. Then you find out it keeps every conversation on disk, answer keys included. Is run 200 quietly reading run 199?

The Code Snippets 403 That Comes From Your Host’s Firewall, Not WordPress

You change a few lines in a snippet, press Save, and get "Could not update snippet. Request failed with status code 403". Press Save on the unchanged snippet and it works. Your login is fine. Something between you and WordPress is reading the code itself.

Why Multisite Subsite Registration Sends Visitors to the Main Site

You put a "create an account" link on one site of a WordPress multisite. Visitors who click it land on the main site, which either says "Registration has been disabled" or signs them up there instead.

Why Your Pre-Push Hook Gets Slower Every Week, and What Fixes It

Every feature arrives with its tests. Nobody times the hook. Then one day git push takes 25 minutes for a one-line change, and the suite is still running one test at a time on a 16-core machine.