Profile Before You Blame: The Slow Step Was a Regex, Not the PDF

A catalog job crawled for the better part of an hour, and the obvious suspect — heavy PDF parsing — took the blame. Profiling put ~40 of every ~41 seconds somewhere no one had looked: a one-line regex hunting for an email that wasn’t there.

Catastrophic backtracking (the pathological end of it is called ReDoS, regular-expression denial of service) is when a regex engine, failing to match, tries an explosive number of ways to match before it gives up. A pattern with a greedy, unbounded run followed by something that often isn’t present forces the engine to advance, fail, back up one character, try again — at every starting position in the input. On a short string you never notice. On a big one, with no match to find, it can burn tens of seconds on a line that looks completely harmless.

The hour that vanished into the wrong suspect

The setup was ordinary. A catalog-enrichment step (“fold”) walked every course, re-derived its summary, category, and tags from a captured web page, and — for one provider — also pulled text out of an attached module PDF. During a whole-catalog run, that provider’s records ground on and on: the job spent close to an hour on them and finished no faster on a re-run.

The story wrote itself. This provider was the one with the PDFs. PDF text extraction is famously heavy. So the working theory, written into the incident notes at the time, was “PDF text is being re-extracted on every fold, ~20 seconds per record.” A fix was even scoped for it: cache the extracted text so re-folds skip the work. Reasonable, plausible, and — as it turned out — beside the point.

The most expensive-looking step is a magnet for blame. That’s exactly why it deserves the least trust until you measure.

Why the PDF was such a convincing scapegoat

Two things made PDF handling the natural culprit, and both are worth naming because they’re the general shape of a misdiagnosis:

• It was the visibly heavy operation. Between “parse a PDF” and “run a small regex over some HTML,” anyone would bet on the PDF. Intuition ranks suspects by how expensive they sound, not by what they cost.
• There was a real, separate bug on the same part. That provider’s PDF handling genuinely was broken in another way — a fraction of its “PDFs” were tiny stylesheet files stored under the wrong content type, and a fix for that was already in flight. When one real defect is sitting on the operating table, every nearby symptom looks like it belongs to the same patient. It didn’t.

That second point is the trap that catches careful people. It isn’t that the team was lazy about evidence — it’s that a legitimate, unrelated bug in the neighborhood lent the wrong theory a borrowed credibility.

Profiling ended the argument in about a minute

The whole misattribution survived only because nobody had run a profiler yet. One cProfile pass on a single record settled it immediately:

• re.search — ~40.3 seconds across 3 calls (~13.5 seconds each).
• PDF text extraction — ~1 second.

The regex was the record. The PDF was a rounding error. No amount of caching PDF text would have moved the ~41-second-per-record number, because the PDF was never the ~41 seconds.

A profiler is cheaper than an argument, and it’s right more often.

Why a “find the email” regex can take 40 seconds

Here is the shape of the offending pattern, generalized — it hunted for an author email on a specific domain, anywhere in the page HTML:

# hunt for an author email anywhere on the page
m = re.search(r"([\w.\-]+@example\.org)", html)Code language: Python (python)

That looks completely innocent, and on a normal string it is. The problem is the interaction of three facts:

1. The input was an ~870 KB rendered page — a lot of ground for a regex to cover.
2. [\w.\-]+ is a greedy, unbounded run of word-ish characters, and HTML is full of long word-character runs.
3. Most pages had no @example.org email on them at all.

So at essentially every position in that 870 KB, the engine greedily consumed a long local-part run, looked for @example.org, didn’t find it, backed up one character, looked again, backed up again — and then moved to the next start position and did the whole thing over. That’s roughly O(n²) work, and the worst case is precisely the no-match case, which was the common case. The pattern spent its time proving a negative, slowly, over and over.

The fix: anchor on the literal, then walk back

The rewrite doesn’t make the regex cleverer — it makes it stop guessing. The one thing that must be present for a match is the literal domain, so scan for that in a single linear pass and only then reconstruct the local part by walking backward:

# anchor on the literal that must be present, then walk back over the local part
author = None
for m in re.finditer(r"@example\.org", html):   # one linear scan, no backtracking
    s = m.start()
    while s > 0 and (html[s - 1].isalnum() or html[s - 1] in "._-"):
        s -= 1
    if s < m.start():                            # a non-empty local part
        author = html[s:m.end()]
        break
Code language: Python (python)

Same output — the same email where one exists, nothing where none does — but the engine never explores dead ends, because it only ever looks at the neighborhood of a literal that’s actually in the text. Per-call time went from ~13.5 seconds to ~0.002 seconds: about 20,000×. The whole-catalog fold for that provider dropped from ~90 minutes to about a minute.

The generalizable move: make the regex anchor on the token that must exist, not on the greedy run that might not lead anywhere. When a pattern’s speed depends on an unbounded quantifier reaching a literal that’s frequently absent, you’ve built a machine for proving negatives the expensive way.

The other fixes were real — and none of them was the win

This is the part that’s easy to skip and shouldn’t be. The PDF-text cache that was scoped early did get built, and it’s a genuine improvement — re-folds skip re-extraction. A second improvement added a “tags-only” fast path so a tag-governance run wouldn’t touch PDFs or summaries at all. Both are correct. Both make the system better. Neither one was the bottleneck.

That’s the subtle cost of a misdiagnosis: it doesn’t only send you down the wrong road, it rewards you with real-feeling progress along the way. You cache the PDF text, the code is genuinely cleaner, a re-fold is faster in some path — and the headline number barely moves, because you optimized a one-second operation while a forty-second one sat untouched. Shipping true-but-secondary fixes feels like solving the problem. Profiling is what tells you whether you actually did.

The short version

A slow step invites you to blame whatever looks most expensive, and PDF parsing looked the part — especially with a real, unrelated PDF bug already being fixed on the same provider. But a one-minute profile put ~40 of ~41 seconds per record on a single regex catastrophically backtracking over an ~870 KB page while searching for an email that usually wasn’t there; the PDF cost ~1 second. Anchoring the pattern on the literal domain and walking back for the local part removed the backtracking entirely — ~20,000× on that call, ~90 minutes down to ~1 for the whole catalog. Two other fixes shipped along the way were real but secondary. The durable habit is the boring one: profile before you optimize, distrust the expensive-looking suspect, and be especially skeptical when a nearby unrelated bug makes the wrong theory feel plausible. (For kin, see why a Postgres connection limit isn’t a concurrency limit and why CI minutes track job count, not build speed — both cases where the number that mattered wasn’t the number people assumed.)

Frequently Asked Questions

What is catastrophic backtracking?

It’s when a regex engine, trying and failing to match, explores an explosive number of possibilities before giving up. A greedy unbounded run (like [\w.\-]+) followed by something that’s often absent forces the engine to advance, fail, back up a character, and retry — at every start position. On large inputs with no match, that quadratic (or worse) work can take tens of seconds. Its weaponized form is called ReDoS.

Why is the “no match” case the slowest?

Because a match lets the engine stop early. When there’s nothing to find, it has to exhaust every path at every position before concluding “no” — which is the maximum amount of backtracking. Patterns are often tested on inputs that do match, so the expensive case hides until real, non-matching data hits it.

How do I find a slow regex like this?

Profile the slow operation — cProfile in Python, or your language’s equivalent — and read where the time actually goes. Don’t reason about it; measure it. A one-minute profile here overturned a diagnosis that had stood in the incident notes for a day.

How do I fix catastrophic backtracking without breaking the match?

Anchor the pattern on a literal that must be present and scan for that first (one linear pass), then reconstruct the surrounding parts — as in the walk-back example above. Other options: bound the quantifier ({1,64} instead of +), make it possessive or atomic if your engine supports it, add a length guard before running the regex, or replace the regex with a plain string scan when the structure is simple enough.

Why did PDF parsing get blamed if it was fast?

Because it was the operation that looked heavy, and because a genuine, separate PDF bug was already being fixed on the same provider — so the slowness had a ready-made, credible-seeming owner. Proximity to a real bug is exactly what makes a wrong theory persuasive. Measurement is the antidote.

Should I have skipped the PDF cache, since it wasn’t the bottleneck?

Not necessarily — the cache is a real improvement and worth keeping. The mistake wasn’t building it; it was believing it would fix the slowness. Ship secondary improvements on their own merits, but don’t let them stand in for the fix the profiler actually points to.