Why a Test That Never Fails Might Be Testing Nothing

Date:

Every check green, a bug still in the running system, and nothing in the suite to explain it. The tests were not wrong about anything. Three of them had never run the check they were named after.

The short answer: a test that never fails is not automatically a good test, because passing is silent. A test earns your trust only once you have watched it go red for the right reason. Four shapes account for most of the quiet ones: a test that reads your source code instead of running it, a test whose setup stops before the branch it is named after, a patch applied to a seam the code under test never reads, and an assertion that holds only because of the machine it ran on. One move separates all four from a real test: break the behavior on purpose, run that single test, and insist on red. If it stays green, the test was never looking.

A test that never fails: what green actually told us

The suite in question covered a small guard in a shell script. The script had to keep exactly one watcher process per log file, so it checked whether a watcher was already following that log before starting another:

# deploy.sh - is a watcher already following this log?
ps -ef | grep -F -- "--log $1" | grep -v grep | head -1Code language: Bash (bash)

Four tests covered that guard. All four passed, every run, for weeks. Meanwhile the machine accumulated eight watcher processes where the guard was supposed to hold it at four.

The guard had never worked once. The script built the log path one way and the process table printed it another, so the two spellings never matched and the guard always reported “no watcher running”. The tests had not noticed, because of what they actually asserted:

def test_the_guard_matches_a_running_watcher():
    src = open("scripts/deploy.sh").read()
    assert "--log" in src
Code language: Python (python)

Green is the absence of a failure, not the presence of a check.

That is the whole problem in one example. Nothing about the test was broken. It answered a different question from the one its name promised, and because it answered that question correctly, it never complained.

The only way to know a test worksBreak the behavior the test is named after, then watch what the test does about it.
1. Break it

Change the code or data the test is named after. One change at a time.

2. Run that test alone

No sibling tests, no cached result. Just the one check you are interrogating.

3. Read the color

Red: the test is real. Restore and move on. Green: it never reached the behavior at all.

green here is the bug

A failure announces itself; a pass says nothing. Until you have seen a test go red for the right reason, it is a claim rather than a check.

The rest of this page is the four shapes, in the order they are worth checking, each with what you see and how to prove it.

The test that reads your source instead of running it

The easiest one to find, and the most common. The test opens a file, or inspects a function’s source, and asserts that some text is present:

assert "timeout" in inspect.getsource(fetch)
assert "--log" in open("scripts/deploy.sh").read()Code language: Python (python)

What you see: nothing. It passes, and it will keep passing.

Why it passes forever: the string is there. It is also still there when the line is commented out, when the flag is spelled correctly in the source and wrongly at the call site, when the branch holding it is unreachable, and when the whole function body has been deleted around it.

How to prove it: empty the function the test claims to cover and run the test again. On the suite above, replacing the guard’s entire body with a bare return left all four tests green. That is the proof, and it takes about twenty seconds.

A source assertion is a reasonable lint rule, and a few of them are genuinely useful: no print in this module, every migration declares a down step. The failure is naming one after a behavior. If a test’s name says the guard matches a running watcher, the test has to run the guard against a process table and read what comes back.

The test whose setup never reaches the branch

Harder, because the test really does execute the code. It just stops short.

The case here was a retry. A lookup picks a set of candidates, and on a collision it is supposed to discard the stale entry and retry exactly once. The test asserted on the value the function returned.

That candidate set was derived from the machine’s own name, matched against a list of known owners. On a machine absent from the list, the candidate set came back empty, the function returned early, and the retry branch was never entered. The returned value was the same either way, so the assertion held. The test passed without ever reaching the retry it was named after.

What you see: a pass, on every machine, including the one where the branch cannot run.

How to prove it: assert that the branch was entered, not only that the answer came out right. A call count, a recorded log event, or a spy on the thing the branch is supposed to do:

def test_the_lookup_retries_once_after_a_collision(lookup, spy):
    lookup.candidates = [0]          # pin the input so the branch must run
    lookup.acquire()
    assert spy.reclaims == 1         # the branch ran, not just "the answer was right"
Code language: Python (python)

That lookup.candidates = [0] line is the real repair. Letting the test inherit the value from its environment is what left the branch unvisited; pinning it forces the code down the path the test’s name claims.

The patch that misses the seam the code uses

The subtlest of the four, and the one that cost the most time.

A test needed the code to believe it was running on a different machine, so it patched what looked like the right seam:

monkeypatch.setattr(mod, "_own_host", lambda: "other-machine")Code language: Python (python)

The function under test did not read mod._own_host. It imported socket inside its own body and called socket.gethostname() directly. The patched attribute was read in exactly one place: the log line the function emits on its way through.

So the output changed. The log said other-machine, which read as proof that the patch had landed, and the computation underneath never saw it. The test reported green on a case it had been written specifically to reproduce.

The patch moved the log line, so the output looked right and the computation never saw it.

How to prove it: patch where the value is looked up, not where it is re-exported. The Python reference calls this where to patch, and the rule is to patch the name in the namespace the code under test resolves at call time. A function doing import socket then socket.gethostname() resolves through the socket module, so that is the only seam that works:

import socket
monkeypatch.setattr(socket, "gethostname", lambda: "other-machine")Code language: Python (python)

The pytest monkeypatch how-to is worth reading end to end before writing a fixture you intend to rely on. More useful than either reference: once the fixture exists, delete it and confirm the test goes red. A fixture that changes nothing is indistinguishable from one that works, as long as the code was already passing.

The test that only passes on your machine

The last shape is the one most people have met, though usually from the other direction: the test that fails in continuous integration and passes at your desk. It is worth looking at from this side, because the green is the more dangerous half.

Two assertions in the same suite depended on the machine sitting on a particular network. On the development machine they passed. They were not reporting anything about the code’s logic; they were reporting that this laptop was on that subnet. The same assertion on any other machine measured something else entirely, and the first machine to disagree was a build runner.

How to prove it: run the one test with the environmental fact changed, not the code. Move the machine, or patch the data the check reads so the condition is false, and confirm the test fails for the stated reason. A test that passes with the condition both true and false is not testing the condition.

The complementary failure, where a hidden dependency bites on a server instead, is worth reading alongside this one: works locally, fails on the server sorts those by symptom.

A checklist before you trust a green test

Run these against any test you are about to rely on, and against every test you write for a bug you just fixed:

  1. Have I watched it fail? If not, you hold a claim rather than a check. Break the behavior, run that one test, require red. The formal version is mutation testing, which does it mechanically by editing your code and counting which tests notice.
  2. Does it run the thing, or read about it? A test asserting on source text proves the text exists. Name it after that, or rewrite it to execute.
  3. Does it reach the branch its name promises? Assert the branch was entered, not only that the final answer looked right. Pin the inputs that steer it rather than inheriting them.
  4. Does my fixture change the value the code reads? Patch the namespace the code resolves at call time, then delete the fixture and confirm red.
  5. Would it still pass on a different machine? If the answer depends on a hostname, a network, a clock, a locale or a file that happens to be present, the test is partly measuring the machine.
  6. Does it fail for the right reason? A red test that fails on a typo in its own setup is not evidence either. Read the failure before you accept it.

That last one links back to a broader habit. An operation reporting success is not the same as the operation having happened, which is this mistake in a different register: a success message is not confirmation.

The short version

A failing test tells you something. A passing test, on its own, tells you very little, because a test can pass by never reaching the behavior it is named after. The four quiet shapes are the source-text assertion, the setup that stops before the branch, the patch applied to a seam the code never reads, and the assertion that holds only on one machine. All four fall to the same move, which costs under a minute: break the behavior deliberately, run that single test, and require it to go red. Until you have seen a test fail for the right reason, treat it as a label rather than a check.

Frequently Asked Questions

Doesn’t code coverage already catch this?

No, and the gap is exactly the one this page is about. Coverage records which lines ran; it says nothing about whether an assertion constrained them. A test that calls your function and asserts nothing at all reports full coverage of everything it touched. The coverage.py FAQ is explicit that the measurement is execution, not correctness. Coverage is good at the opposite question, finding the branch nothing visits: the retry case above would show up in a branch-coverage report as an unvisited branch, which is a strong reason to read one.

Should I run a mutation testing tool across the whole suite?

Not as a first move. A full mutation run is slow, and on a suite of a few hundred tests it produces a long list of surviving mutants nobody works through. Point it at the module you just changed, or do the one-line version by hand on the specific test you are about to depend on. The habit matters more than the tooling, since the manual break-it-and-look loop catches the same four shapes in a minute.

How do I check a test I inherited and cannot easily break?

If reaching the behavior is hard enough that you cannot make the test fail, that is a finding about the code rather than the test, and usually the same reason the test was written shallowly to begin with. Two things work short of a refactor. Run the test with its fixtures removed one at a time and see whether any of them were load-bearing. Then read the assertion against the test’s name as if a stranger wrote both, and where the name promises more than the assertion delivers, rename the test to what it checks today. A test honestly called test_the_guard_source_mentions_the_log_flag is a small, useful thing. The same test named after the behavior is a liability, because it will be counted as cover for something nobody is watching.

Share post:

Subscribe

Popular

More like this
Related

Can AI CLI Conversation History Leak Answers Into a Blind Test?

You use an AI command-line tool to grade or answer hundreds of questions, one fresh run each. Then you find out it keeps every conversation on disk, answer keys included. Is run 200 quietly reading run 199?

The Code Snippets 403 That Comes From Your Host’s Firewall, Not WordPress

You change a few lines in a snippet, press Save, and get "Could not update snippet. Request failed with status code 403". Press Save on the unchanged snippet and it works. Your login is fine. Something between you and WordPress is reading the code itself.

Why a Flaky Test Passes Alone and Fails When the Machine Is Busy

Run the test by itself and it passes, every time. Run the whole suite in parallel and it fails, but only some days, and its failure looks exactly like a real bug. In our case the code was fine. The test was waiting on a clock.

Why Multisite Subsite Registration Sends Visitors to the Main Site

You put a "create an account" link on one site of a WordPress multisite. Visitors who click it land on the main site, which either says "Registration has been disabled" or signs them up there instead.