Your scraper runs clean every night and the records look complete. Then someone notices that out-of-stock books were listed as buyable for weeks, because the page said so on a button the parser never read.
Saved pages, the current parser. No network, no database writes.
Card text blocks, short values and attributes, the page outside cards, stored vs re-parse.
In the record: ok. In the register: left out, with a reason. Neither: a finding.
Exit 1 on anything unregistered. Run it after every parser change.
Short values (a price, a stock label, an attribute) are where silent drops hide; a check of long text blocks alone misses them.
Table of Contents
The scraper parser passes its tests and still drops a field
Take a bookstore’s category page. Each product card shows a cover, a title, an author, a price, a rating and an “Add to Cart” button. A parser reads the cover, title, author, price and rating, and its tests pass. On 268 of 3,007 cards in the test crawl (9%), the button said “Notify Me” instead. That label was the stock state, and it never reached a single record.
Nothing failed. The parser did exactly what it was written to do, and its tests checked exactly what it returned. A test asserts on the values a parser produces; nothing asserts on the values the page held and the parser never touched. That is the same blind spot as a test that never fails: a green result that never reached the behavior it is named after.
A quick word-overlap check does not catch it either. The words of a dropped block often appear somewhere else in the record, so “most of the page’s words are in the record” stays true while a whole field is missing.
A parser never reports the field it did not read, so the only way to see it is to account for the page.
Step 1: Re-parse the pages you saved, not the live site
You can only audit what you kept. Save every page the crawler fetches, compressed, next to the run that fetched it. The audit then replays those files through the parser as it is today: no network, no database writes, and the same input every time. When you fix the parser, you run the audit again on the same pages and watch the finding disappear.
Saved pages pay a second time. Once the parser keeps a field it used to drop, a re-parse of the old pages recovers that field for every product already crawled, with no new requests to the site.
If the crawler drives a browser and saves its own extract (a JSON of what the page script returned), save the raw page beside it too. Your extract is already a parse, so auditing it only checks one parser against another.
Step 2: Check every text block of a card against that card’s record
Cut the page into product cards the same way the parser does; the simplest route is to have the parser expose its card cutter. Then split each card into blocks: the text that each block element (a paragraph, a list item, a heading, a div) holds itself. Check each block against that card’s own record, not the whole page’s, and give it one of four verdicts:
- whole: the block’s words appear in the record, in order;
- cleaned: at least 90% of them do and the end is intact, so a cleaner removed something small;
- partial: only part of them do, because the value was cut short or merged;
- missing: none or almost none of them do.
Group the results by where each block sits, its element path (such as div.award-grid > p), so one parser gap is one finding across thousands of cards. On the bookstore this step found award lines (“1961 | Pulitzer Prize for Fiction”) on 30 books: 63 blocks, one finding.
Blocks under three words are not judged here, because a two-word block could be a label or a value and the check cannot tell which. That gap is where the next step comes in.
Step 3: Check the short values one by one, attributes included
Prices, ratings, stock labels and ids are short, and many of them live in attributes rather than visible text. So list every element’s own text and every attribute value in the card, each with a name such as button.cart-btn#text or img@alt, and look each value up in the record. Normalize both sides to lowercase word tokens and match whole tokens, with the record flattened so its keys and numbers count too.
import re
from html.parser import HTMLParser
SKIP = {"class", "style", "width", "height", "type"}
VOID = {"img", "br", "input", "hr", "meta", "link", "source", "wbr"}
def norm(s):
"""Lowercase word tokens; a thousands comma between digits is joined ('1,554' -> '1554')."""
return " ".join(re.findall(r"[0-9a-z]+", re.sub(r"(?<=\d),(?=\d)", "", str(s).lower())))
class CardFields(HTMLParser):
"""(name, value) for every element's own text and every attribute value in one card."""
def __init__(self):
super().__init__(convert_charrefs=True)
self.stack, self.out = [], []
def handle_starttag(self, tag, attrs):
a = dict(attrs)
name = tag + ("." + a["class"].split()[0] if (a.get("class") or "").split() else "")
self.out += [(f"{name}@{k}", v) for k, v in a.items() if k not in SKIP and v]
if tag not in VOID:
self.stack.append((tag, name))
def handle_endtag(self, tag):
for i in range(len(self.stack) - 1, -1, -1):
if self.stack[i][0] == tag:
del self.stack[i:]
return
def handle_data(self, data):
text = " ".join(data.split())
if text and self.stack:
self.out.append((self.stack[-1][1] + "#text", text))
def lost_fields(card_html, record_text):
"""Values in the card whose tokens are not in the record. record_text: norm() of the whole record, space-padded."""
p = CardFields()
p.feed(card_html)
return [(n, v) for n, v in p.out if norm(v) and f" {norm(v)} " not in record_text]
Code language: Python (python)
On the four test sites, this step found what the block check could not:
- A stock state on a button. “Notify Me” in the button’s text and its
title, on 9% of the bookstore’s cards. - A seller id inside a click handler. Every card’s button carried a call like
AddtoCart('48213907', 1, '355', 1). The first argument was the product id the parser already kept. Rather than guess the rest from its shape, read the site’s own listing script: it defined the function asAddtoCart(productId, quantity, vendorId, productTypeId). So355named a seller, and the crawl held 61 different ones. - Image alt text carrying the brand. On a marketplace, 4,484 cards had alt text the record did not hold, often with the brand name the visible title left out.
- Color swatch links to sibling products. A few cards linked to other product ids of the same family through data attributes: 33 links on 11 cards, exactly the input a “group these variants” feature needs.
Name a value from the site’s own code, not from its shape; a guessed field name is a guess stored forever.
Keep generic “chrome” rules away from short values
Most audits start with a default rule that buttons, forms and labels are page furniture. For a page’s blocks that rule is right. Applied to a card’s fields, it hid the stock state, because a card’s button is precisely where a store puts “Notify Me”. Register the exact value you mean to ignore, with a pattern on the value itself: the label Add to Cart only, so that Notify Me on the same attribute still fails.
Step 4: Account for the page outside the cards
Remove the card markup from the page, split what remains into blocks, and check them against every card record on the page together. Expect filters with counts, the category menu, a result total, the category’s SEO description and, on large stores, other products shown outside the result grid: sponsored banners and “related to your search” strips. Each one is a decision: register it with its reason, or treat it as something the crawler should collect.
Text that appears identically on a large share of the pages, here 30% of them and at least ten, is the site’s frame (header, footer, pop-ups) and is set aside automatically. Watch that threshold; it caused one of the false positives below.
Step 5: Compare the stored record with a re-parse of the same run
The first four steps test today’s parser. This one tests the data you already stored. For each stored record, re-parse the page from the run that wrote it and compare the two. Use that same run, not the newest page: a record a later crawl touched holds that later visit’s values, and comparing it with an older page reports a difference that is not there.
On the bookstore, 2,045 of 2,058 stored records matched their re-parse exactly. The other 13 were the store’s own duplicates: the same ISBN listed two or three times on one page under consecutive internal ids, same title, price and seller. The record is keyed by ISBN, so they collapsed into one. That is not a parser fault, and the right response is to detect it and report it, never to rewrite the store’s data to look tidy. On one marketplace, 366 of 5,891 records differed only in URL tracking parameters.
Write a register of what you leave out on purpose
Not every value belongs in the record: a lazy-load placeholder image, a broken-image script, a “No Review Yet” label that the record already expresses as a missing rating. Each of those goes into a register, one entry per decision, with the reason written down:
{"entries": [
{"signature": "^field:img\\.book-cover@src$", "text": "lazy-loading\\.gif$",
"kind": "not-content", "reason": "Lazy-load placeholder. The real cover is in data-src, which is kept."},
{"signature": "^field:button\\.cart-btn@title$", "text": "^Add to Cart$",
"kind": "not-content", "reason": "The label of a card that can be bought. Only this label: 'Notify Me' is a stock state."}
]}
Code language: JSON / JSON with Comments (json)
Three rules keep the register honest. Add an entry only after reading the blocks it covers. Never register a real product fact the parser happens to miss: it stays a finding until the parser keeps it. And keep entries in step with the parser, because an entry that matches more than you meant hides a drop.
The gate is one line of logic: exit 1 when cards, fields or the page hold any loss that is neither in the record nor in the register. Run it after every parser change. A decision you deferred (“not collecting that marker for now”) stays unregistered on purpose, so the gate keeps reminding you it is open.
A register entry says “left out on purpose, because…”; a missing entry says nobody decided.
The audit’s own false positives
The first runs flagged values that had in fact reached the record. Each came from representation, and each was fixed by making the comparison canonical, never by loosening the match:
- Thousands commas. The page printed
1,554; the record held1554. Join a comma between two digits before comparing. 4.0against4. The record stored a rating as the float4.0; the page printed4. Render a whole-number float both ways.- Text printed twice. A card printed the same text twice, and the record kept it once. Register the repeat with that reason.
- A value spread over two text nodes. A record that keeps a card’s text node by node holds “FREE delivery” and “Thu, 12 Nov” as two strings. Also check each list of strings joined in its order.
- A site frame that fell below the template share. Once the crawl went deep, most saved pages were “load more” replies carrying cards and no header or footer. The frame then appeared on under 30% of pages and surfaced as hundreds of findings. Register the frame explicitly, with that reason.
Loosening the match (“80% of the words is close enough”) would have silenced all five, and every real drop with them.
What the audit found on four stores
| Site | Pages / cards | First run | After the parser fix |
|---|---|---|---|
| Bookstore | 156 / 3,007 | Awards, stock button, seller id, one unexplained badge | 275 pages / 5,387 cards: no findings |
| Marketplace A | 210 / 6,766 | Image alt with brand, swatch links, a storefront marker, one customizable product | Alt and swatches kept; the marker deferred, so the gate still fails on it |
| Marketplace B | 208 / 7,932 | Price, list price and discount glued into one string; no product image; a badge | Text kept node by node, images kept |
| Electronics retailer | 652 / 13,557 | No findings | Nothing to fix |
The electronics retailer came out clean for a simple reason: its listing data arrives as a JSON object per product, and the parser stores that object whole. When the source is already structured, keeping it verbatim beats picking fields out of it.
Marketplace B’s finding was a different kind of loss: the parser kept the text, but the browser had glued the values together before the parser ever saw them, which is how innerText reads inline elements.
The short version
Tests check what a scraper parser returns, so they cannot see what it skipped. Save the raw pages, re-parse them with the current parser, and account for every block, every short value and every attribute in each card, the page outside the cards, and the stored records against a same-run re-parse. Write what you leave out on purpose into a register with reasons, fix the comparison when it raises false positives, and gate on anything unregistered.
Frequently Asked Questions
Will the audit catch a value the parser reads wrongly?
No. It checks that a value from the page reached the record, not that it landed under the right name. A seller id stored as the price would pass. Pair the audit with a few field-level tests that check known cards’ values in their proper fields.
How often should I run it?
After every parser change, as a gate, and from time to time on freshly saved pages. Sites add fields and move values without notice, and a new field shows up as a new finding on the next run. A store you automate without owning it can change under you at any time.
What about pages built by JavaScript?
Audit what the browser rendered, saved as HTML, rather than your crawler’s own extract of it. If the crawler saves a JSON of selected values, that file is already a parse, so keep the rendered page beside it and audit the cards from the page.