Every check green, a bug still in the running system, and nothing in the suite to explain it. The tests were not wrong about anything. Three of them had never run the check they were named after.
Table of Contents
A test that never fails: what green actually told us
The suite in question covered a small guard in a shell script. The script had to keep exactly one watcher process per log file, so it checked whether a watcher was already following that log before starting another:
# deploy.sh - is a watcher already following this log?
ps -ef | grep -F -- "--log $1" | grep -v grep | head -1Code language: Bash (bash)
Four tests covered that guard. All four passed, every run, for weeks. Meanwhile the machine accumulated eight watcher processes where the guard was supposed to hold it at four.
The guard had never worked once. The script built the log path one way and the process table printed it another, so the two spellings never matched and the guard always reported “no watcher running”. The tests had not noticed, because of what they actually asserted:
def test_the_guard_matches_a_running_watcher():
src = open("scripts/deploy.sh").read()
assert "--log" in src
Code language: Python (python)
Green is the absence of a failure, not the presence of a check.
That is the whole problem in one example. Nothing about the test was broken. It answered a different question from the one its name promised, and because it answered that question correctly, it never complained.
Change the code or data the test is named after. One change at a time.
No sibling tests, no cached result. Just the one check you are interrogating.
Red: the test is real. Restore and move on. Green: it never reached the behavior at all.
A failure announces itself; a pass says nothing. Until you have seen a test go red for the right reason, it is a claim rather than a check.
The rest of this page is the four shapes, in the order they are worth checking, each with what you see and how to prove it.
The test that reads your source instead of running it
The easiest one to find, and the most common. The test opens a file, or inspects a function’s source, and asserts that some text is present:
assert "timeout" in inspect.getsource(fetch)
assert "--log" in open("scripts/deploy.sh").read()Code language: Python (python)
What you see: nothing. It passes, and it will keep passing.
Why it passes forever: the string is there. It is also still there when the line is commented out, when the flag is spelled correctly in the source and wrongly at the call site, when the branch holding it is unreachable, and when the whole function body has been deleted around it.
How to prove it: empty the function the test claims to cover and run the test again. On the suite above, replacing the guard’s entire body with a bare return left all four tests green. That is the proof, and it takes about twenty seconds.
A source assertion is a reasonable lint rule, and a few of them are genuinely useful: no print in this module, every migration declares a down step. The failure is naming one after a behavior. If a test’s name says the guard matches a running watcher, the test has to run the guard against a process table and read what comes back.
The test whose setup never reaches the branch
Harder, because the test really does execute the code. It just stops short.
The case here was a retry. A lookup picks a set of candidates, and on a collision it is supposed to discard the stale entry and retry exactly once. The test asserted on the value the function returned.
That candidate set was derived from the machine’s own name, matched against a list of known owners. On a machine absent from the list, the candidate set came back empty, the function returned early, and the retry branch was never entered. The returned value was the same either way, so the assertion held. The test passed without ever reaching the retry it was named after.
What you see: a pass, on every machine, including the one where the branch cannot run.
How to prove it: assert that the branch was entered, not only that the answer came out right. A call count, a recorded log event, or a spy on the thing the branch is supposed to do:
def test_the_lookup_retries_once_after_a_collision(lookup, spy):
lookup.candidates = [0] # pin the input so the branch must run
lookup.acquire()
assert spy.reclaims == 1 # the branch ran, not just "the answer was right"
Code language: Python (python)
That lookup.candidates = [0] line is the real repair. Letting the test inherit the value from its environment is what left the branch unvisited; pinning it forces the code down the path the test’s name claims.
The patch that misses the seam the code uses
The subtlest of the four, and the one that cost the most time.
A test needed the code to believe it was running on a different machine, so it patched what looked like the right seam:
monkeypatch.setattr(mod, "_own_host", lambda: "other-machine")Code language: Python (python)
The function under test did not read mod._own_host. It imported socket inside its own body and called socket.gethostname() directly. The patched attribute was read in exactly one place: the log line the function emits on its way through.
So the output changed. The log said other-machine, which read as proof that the patch had landed, and the computation underneath never saw it. The test reported green on a case it had been written specifically to reproduce.
The patch moved the log line, so the output looked right and the computation never saw it.
How to prove it: patch where the value is looked up, not where it is re-exported. The Python reference calls this where to patch, and the rule is to patch the name in the namespace the code under test resolves at call time. A function doing import socket then socket.gethostname() resolves through the socket module, so that is the only seam that works:
import socket
monkeypatch.setattr(socket, "gethostname", lambda: "other-machine")Code language: Python (python)
The pytest monkeypatch how-to is worth reading end to end before writing a fixture you intend to rely on. More useful than either reference: once the fixture exists, delete it and confirm the test goes red. A fixture that changes nothing is indistinguishable from one that works, as long as the code was already passing.
The test that only passes on your machine
The last shape is the one most people have met, though usually from the other direction: the test that fails in continuous integration and passes at your desk. It is worth looking at from this side, because the green is the more dangerous half.
Two assertions in the same suite depended on the machine sitting on a particular network. On the development machine they passed. They were not reporting anything about the code’s logic; they were reporting that this laptop was on that subnet. The same assertion on any other machine measured something else entirely, and the first machine to disagree was a build runner.
How to prove it: run the one test with the environmental fact changed, not the code. Move the machine, or patch the data the check reads so the condition is false, and confirm the test fails for the stated reason. A test that passes with the condition both true and false is not testing the condition.
The complementary failure, where a hidden dependency bites on a server instead, is worth reading alongside this one: works locally, fails on the server sorts those by symptom.
A checklist before you trust a green test
Run these against any test you are about to rely on, and against every test you write for a bug you just fixed:
- Have I watched it fail? If not, you hold a claim rather than a check. Break the behavior, run that one test, require red. The formal version is mutation testing, which does it mechanically by editing your code and counting which tests notice.
- Does it run the thing, or read about it? A test asserting on source text proves the text exists. Name it after that, or rewrite it to execute.
- Does it reach the branch its name promises? Assert the branch was entered, not only that the final answer looked right. Pin the inputs that steer it rather than inheriting them.
- Does my fixture change the value the code reads? Patch the namespace the code resolves at call time, then delete the fixture and confirm red.
- Would it still pass on a different machine? If the answer depends on a hostname, a network, a clock, a locale or a file that happens to be present, the test is partly measuring the machine.
- Does it fail for the right reason? A red test that fails on a typo in its own setup is not evidence either. Read the failure before you accept it.
That last one links back to a broader habit. An operation reporting success is not the same as the operation having happened, which is this mistake in a different register: a success message is not confirmation.
The short version
A failing test tells you something. A passing test, on its own, tells you very little, because a test can pass by never reaching the behavior it is named after. The four quiet shapes are the source-text assertion, the setup that stops before the branch, the patch applied to a seam the code never reads, and the assertion that holds only on one machine. All four fall to the same move, which costs under a minute: break the behavior deliberately, run that single test, and require it to go red. Until you have seen a test fail for the right reason, treat it as a label rather than a check.
Frequently Asked Questions
Doesn’t code coverage already catch this?
No, and the gap is exactly the one this page is about. Coverage records which lines ran; it says nothing about whether an assertion constrained them. A test that calls your function and asserts nothing at all reports full coverage of everything it touched. The coverage.py FAQ is explicit that the measurement is execution, not correctness. Coverage is good at the opposite question, finding the branch nothing visits: the retry case above would show up in a branch-coverage report as an unvisited branch, which is a strong reason to read one.
Should I run a mutation testing tool across the whole suite?
Not as a first move. A full mutation run is slow, and on a suite of a few hundred tests it produces a long list of surviving mutants nobody works through. Point it at the module you just changed, or do the one-line version by hand on the specific test you are about to depend on. The habit matters more than the tooling, since the manual break-it-and-look loop catches the same four shapes in a minute.
How do I check a test I inherited and cannot easily break?
If reaching the behavior is hard enough that you cannot make the test fail, that is a finding about the code rather than the test, and usually the same reason the test was written shallowly to begin with. Two things work short of a refactor. Run the test with its fixtures removed one at a time and see whether any of them were load-bearing. Then read the assertion against the test’s name as if a stranger wrote both, and where the name promises more than the assertion delivers, rename the test to what it checks today. A test honestly called test_the_guard_source_mentions_the_log_flag is a small, useful thing. The same test named after the behavior is a liability, because it will be counted as cover for something nobody is watching.