Works Locally, Fails on the Server? Four Hidden Causes and How to Tell Which Is Yours

Date:

The tests are green on your laptop. You push, the same commit runs on a server or a CI runner, and it fails, sometimes with an error, sometimes with nothing at all. You didn’t change a line. So what changed?

The short answer: when code works locally and fails somewhere else, the machine changed, and your code was quietly relying on it. Your own computer supplies things the code never asks for out loud: the line endings your tools expect, a terminal that accepts any character, a config file that is already in the folder, a test database nobody else is writing to. A server or CI runner is often the first machine that doesn’t supply them. Four symptoms cover most of these failures: values that look identical but don’t match, a crash on a harmless log line, a tool that says “not configured” only in CI, and a check that fails under real load but never on your machine. Each has a one-line test that tells you which one you have (the table below), and each has the same fix: state the assumption in the code, and test under the server’s conditions instead of your own.

It works locally but not on the server: which cause is yours?

Find your symptom in the first column, run the check, and jump to that section.

What you see on the server or in CILikely causeCheck it in one line
A comparison fails, though printing both values shows the same textA hidden carriage return (\r) on the end of the valuePipe the output through cat -A: a ^M before each $ means \r
UnicodeEncodeError on a print() or log lineOutput is going to a pipe or file in a legacy encoding, not UTF-8Run it locally with output redirected to a file: python script.py > out.txt
“Not configured” (or a missing key, host or path) only in CIYour machine has a config file or variable the clean runner doesn’tMove the config file aside, unset the variable, run it again
A “copy, then verify” step fails sometimes, only under real trafficThe source changed while the copy was runningRun it against a copy of the source with something writing to it

Values that look identical but don’t match

You have a script that reads another program’s output and acts on it. Here a small Python program prints status=ok, and a Bash loop checks for ok. On a test machine running Windows with Git Bash, the check fails, even though the value printed between the brackets looks right:

$ python status.py | while IFS='=' read -r k v; do echo "[$k] [$v]"; [ "$v" = "ok" ] && echo "matched" || echo "did not match"; done
[status] [ok]
did not matchCode language: PHP (php)

Confirm it. cat -A shows characters that don’t print. A ^M before the end-of-line $ is a carriage return:

$ python status.py | cat -A
status=ok^M$
count=3^M$

Why it happens. Windows ends text lines with \r\n, Unix with \n. Python on Windows writes \r\n when printing. Bash’s read removes the \n and keeps the \r, so the variable holds ok\r, which is not ok. When you print it, the \r just moves the cursor back to the start of the line, so you never see it. On your Mac or Linux machine the producer writes plain \n and the same script works.

The character that breaks the comparison is the one that doesn’t show up when you print it.

Fix it. Remove the carriage return at the point where one tool’s output becomes another tool’s input, not deep in the logic:

$ python status.py | tr -d '\r' | while IFS='=' read -r k v; do [ "$v" = "ok" ] && echo "matched"; done
matchedCode language: PHP (php)

In Python, line.rstrip("\r\n") does the same job on the reading side. A .gitattributes rule won’t help here: it controls line endings of files in the repository, and these \r characters are written by a running program.

A crash on a harmless log line

The script works when you run it by hand. On the server it dies on a progress message, before or after the real work, with this:

$ python report.py > out.txt
Traceback (most recent call last):
  File "report.py", line 3, in <module>
    print("Copied 3 files → backup ✓")
UnicodeEncodeError: 'charmap' codec can't encode character '→' in position 15: character maps to <undefined>Code language: JavaScript (javascript)

Confirm it. Notice the > out.txt. The same command printing straight to the terminal works. Print sys.stdout.encoding to stderr and the difference shows: on the test machine it was cp1252 when redirected. Scheduled tasks, services, remote sessions and CI runners all capture output instead of showing it in a terminal, which is why the crash appears there and not when you run the script yourself.

Why it happens. Since Python 3.6, printing to a real Windows console uses Unicode and doesn’t fail. When output goes to a pipe or a file, Python uses the system’s legacy code page instead, often Windows-1252, which has room for only 256 characters. The arrow (→) and the check mark (✓) aren’t among them, so the first print() that contains one raises an error and stops the program. (The em dash and en dash are in Windows-1252, so a dash alone won’t trigger it, which is why this can hide for a long time.) The calculation was fine. Writing a message about it is what failed.

Your program’s survival shouldn’t depend on which characters a log line happens to contain.

Fix it. Choose the output encoding instead of inheriting it. Set the environment variable PYTHONIOENCODING=utf-8 (or PYTHONUTF8=1) for the job, or call sys.stdout.reconfigure(encoding="utf-8") at the start of the script. With that set, the same redirected run works:

$ PYTHONIOENCODING=utf-8 python report.py > out.txt; cat out.txt
Copied 3 files → backup ✓

If the output is read by other programs, plain ASCII (->, OK) is safer still.

“Not configured,” but only in CI

A tool reads its connection settings from a local config file, or from an environment variable if there is no file, and exits if it finds neither. You run it: it works. Your pre-commit hook runs it: it passes. CI runs the same commit and stops at once:

$ python tool.py
connecting to db.example.com:5432

# in CI
not configuredCode language: PHP (php)

Confirm it. Recreate CI’s empty machine on your own: move the config file out of the folder, unset the variable, and run again.

$ mv app.conf app.conf.aside
$ env -u APP_DB_URL python tool.py
not configured

Why it happens. The config file was in your folder all along (it’s often listed in .gitignore because it holds settings for your machine), so the “neither exists” branch never ran at home. Your pre-commit hook runs on your machine, with your files, so it passed because of the file. A CI runner starts from a fresh checkout with only what’s committed and only the variables you configured for it. Nothing is broken: the tool did exactly what it was written to do.

A check that runs on your machine also gets your machine’s files and settings.

Fix it. Two halves. In the code, make the requirement loud: fail with a message that says which file or variable is missing and how to provide it, not just “not configured.” In your routine, reproduce the empty machine before you merge (the two commands above), and give CI the variable it needs through its secrets or settings rather than a committed file.

A check that fails under load and never on your machine

A job copies a large table out of a live database, then checks the copy by counting the source rows again and comparing. On your machine it passes every time. In production it fails now and then, and when you try to reproduce it, it passes again.

Here is the same job on a test machine, run against a small SQLite database, first with nothing else touching it and then with a second process adding rows during the copy:

$ python copycheck.py naive idle
naive     idle: copied 100000, source now 100000 -> OK
$ python copycheck.py naive live
naive     live: copied 100040, source now 100211 -> MISMATCH

Confirm it. If the failure only appears when something else is writing to the source, run your check against a copy that has a writer running (a loop inserting rows is enough). If it fails there and passes on an idle copy, this is your bug.

Why it happens. “Copy, then count again” is correct for a source that stays still. A busy database doesn’t: rows keep arriving during the minutes the copy takes, so the second count includes rows the copy never saw. Your test database is idle, so it can’t show the problem. Concurrency is the one thing it doesn’t have.

Fix it. Do the copy and the verification against the same point-in-time view of the data. In most databases that means running both inside one transaction with snapshot (repeatable-read) isolation, so both reads see the same moment. In the test, wrapping both reads in one read transaction fixed it with the writer still running:

$ python copycheck.py snapshot live
snapshot  live: copied 100036, source now 100036 -> OK

PostgreSQL goes further: you can export a transaction’s snapshot and pass it to pg_dump --snapshot, so a separate verification step sees exactly the state the dump saw.

The pattern: name the assumption, test where it runs

All four bugs are a gap between something your code assumed and something your machine happened to provide. The fix has the same two steps every time:

  1. Write the assumption into the code. Strip the \r where you read. Set the output encoding. Fail loudly when config is missing. Read the copy and the check from one snapshot. Each change turns an invisible dependency into a stated one.
  2. Test under the server’s conditions, not yours. Before you deploy, redirect the output to a file, run on the target operating system, move the config aside and empty the environment, and run against a source that something is writing to.
Works locally, fails on the serverEach symptom hides behind something your machine supplies and the server doesn't.
Equal, but no match

Values print the same but compare unequal. Cause: a hidden \r. Fix: strip it where you read.

Log line crash

UnicodeEncodeError when output is piped. Cause: legacy encoding. Fix: set UTF-8, or keep output ASCII.

Only fails in CI

"Not configured" on a clean runner. Cause: a file only your machine has. Fix: fail loudly.

Only under load

Copy-then-verify fails sometimes. Cause: source moved during the copy. Fix: one snapshot.

Underneath all four: a green check on your machine is evidence about your machine. Test under the server's conditions.

The short version

When code works locally but fails on the server or in CI, look first for something your machine provides without being asked: Unix line endings, a terminal that takes any character, a config file in the folder, a quiet database. The table at the top maps each symptom to a one-line check. The fix is always to state the assumption in the code and test under the server’s conditions. For individual Windows traps by name, see the field guide of toolchain gotchas; for why a pull request can be mergeable and still fail its checks, see merge conflict vs CI check.

Frequently Asked Questions

Won’t running everything in Docker fix this?

It fixes some of it. A container gives every machine the same operating system, line endings and installed files, which removes the first symptom and most of the third. It doesn’t give you real traffic, so the copy-and-verify bug still only appears in production, and it doesn’t set the output encoding unless the image sets a UTF-8 locale or PYTHONIOENCODING. A config file that you mount into your local container but not in CI is the same bug as before.

Should I set PYTHONUTF8=1 everywhere?

It’s a reasonable default for scripts and services you control, and it makes Python use UTF-8 for files and output whatever the system code page is. Two cautions: set it in the job’s own environment (the CI config, the service definition), not only in your shell profile, or you have recreated the third symptom; and a file that really is in a legacy encoding will now be read wrongly unless you open it with an explicit encoding=. Newer Python versions are moving to UTF-8 by default, but don’t rely on the version on the server matching yours.

My values have no carriage return, but the comparison still fails. What else can hide?

Trailing spaces, a non-breaking space (common in text pasted from a web page or a spreadsheet), a byte-order mark at the start of a file, and tabs instead of spaces. cat -A shows tabs as ^I; for the others, look at the bytes with od -c or xxd, or print repr(value) in Python.

Share post:

Subscribe

Popular

More like this
Related

Can AI CLI Conversation History Leak Answers Into a Blind Test?

You use an AI command-line tool to grade or answer hundreds of questions, one fresh run each. Then you find out it keeps every conversation on disk, answer keys included. Is run 200 quietly reading run 199?

The Code Snippets 403 That Comes From Your Host’s Firewall, Not WordPress

You change a few lines in a snippet, press Save, and get "Could not update snippet. Request failed with status code 403". Press Save on the unchanged snippet and it works. Your login is fine. Something between you and WordPress is reading the code itself.

Why a Flaky Test Passes Alone and Fails When the Machine Is Busy

Run the test by itself and it passes, every time. Run the whole suite in parallel and it fails, but only some days, and its failure looks exactly like a real bug. In our case the code was fine. The test was waiting on a clock.

Why Multisite Subsite Registration Sends Visitors to the Main Site

You put a "create an account" link on one site of a WordPress multisite. Visitors who click it land on the main site, which either says "Registration has been disabled" or signs them up there instead.