How to Audit What a Scraper Parser Silently Drops

Date:

Your scraper runs clean every night and the records look complete. Then someone notices that out-of-stock books were listed as buyable for weeks, because the page said so on a button the parser never read.

The short answer: A scraper parser fails in one direction without a sound: it never reports what it did not read. To find those gaps, keep the raw pages, re-run the current parser on them offline, and account for every piece of each page: each card’s text blocks, each element’s text and attribute values (short ones such as prices and button labels included), the page outside the cards, and the stored record against a re-parse of the same run. Whatever the record lacks is either left out on purpose, written into a register with its reason, or it is a finding. A gate that fails on any unregistered finding keeps it that way after every parser change.
Account for every value on the pageRe-parse saved pages, check each piece against the record, and fail on anything nobody decided.
1. Re-parse

Saved pages, the current parser. No network, no database writes.

2. Account

Card text blocks, short values and attributes, the page outside cards, stored vs re-parse.

3. Sort

In the record: ok. In the register: left out, with a reason. Neither: a finding.

4. Gate

Exit 1 on anything unregistered. Run it after every parser change.

fails on any finding

Short values (a price, a stock label, an attribute) are where silent drops hide; a check of long text blocks alone misses them.

The scraper parser passes its tests and still drops a field

Take a bookstore’s category page. Each product card shows a cover, a title, an author, a price, a rating and an “Add to Cart” button. A parser reads the cover, title, author, price and rating, and its tests pass. On 268 of 3,007 cards in the test crawl (9%), the button said “Notify Me” instead. That label was the stock state, and it never reached a single record.

Nothing failed. The parser did exactly what it was written to do, and its tests checked exactly what it returned. A test asserts on the values a parser produces; nothing asserts on the values the page held and the parser never touched. That is the same blind spot as a test that never fails: a green result that never reached the behavior it is named after.

A quick word-overlap check does not catch it either. The words of a dropped block often appear somewhere else in the record, so “most of the page’s words are in the record” stays true while a whole field is missing.

A parser never reports the field it did not read, so the only way to see it is to account for the page.

Step 1: Re-parse the pages you saved, not the live site

You can only audit what you kept. Save every page the crawler fetches, compressed, next to the run that fetched it. The audit then replays those files through the parser as it is today: no network, no database writes, and the same input every time. When you fix the parser, you run the audit again on the same pages and watch the finding disappear.

Saved pages pay a second time. Once the parser keeps a field it used to drop, a re-parse of the old pages recovers that field for every product already crawled, with no new requests to the site.

If the crawler drives a browser and saves its own extract (a JSON of what the page script returned), save the raw page beside it too. Your extract is already a parse, so auditing it only checks one parser against another.

Step 2: Check every text block of a card against that card’s record

Cut the page into product cards the same way the parser does; the simplest route is to have the parser expose its card cutter. Then split each card into blocks: the text that each block element (a paragraph, a list item, a heading, a div) holds itself. Check each block against that card’s own record, not the whole page’s, and give it one of four verdicts:

  • whole: the block’s words appear in the record, in order;
  • cleaned: at least 90% of them do and the end is intact, so a cleaner removed something small;
  • partial: only part of them do, because the value was cut short or merged;
  • missing: none or almost none of them do.

Group the results by where each block sits, its element path (such as div.award-grid > p), so one parser gap is one finding across thousands of cards. On the bookstore this step found award lines (“1961 | Pulitzer Prize for Fiction”) on 30 books: 63 blocks, one finding.

Blocks under three words are not judged here, because a two-word block could be a label or a value and the check cannot tell which. That gap is where the next step comes in.

Step 3: Check the short values one by one, attributes included

Prices, ratings, stock labels and ids are short, and many of them live in attributes rather than visible text. So list every element’s own text and every attribute value in the card, each with a name such as button.cart-btn#text or img@alt, and look each value up in the record. Normalize both sides to lowercase word tokens and match whole tokens, with the record flattened so its keys and numbers count too.

import re
from html.parser import HTMLParser

SKIP = {"class", "style", "width", "height", "type"}
VOID = {"img", "br", "input", "hr", "meta", "link", "source", "wbr"}

def norm(s):
    """Lowercase word tokens; a thousands comma between digits is joined ('1,554' -> '1554')."""
    return " ".join(re.findall(r"[0-9a-z]+", re.sub(r"(?<=\d),(?=\d)", "", str(s).lower())))

class CardFields(HTMLParser):
    """(name, value) for every element's own text and every attribute value in one card."""
    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.stack, self.out = [], []

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        name = tag + ("." + a["class"].split()[0] if (a.get("class") or "").split() else "")
        self.out += [(f"{name}@{k}", v) for k, v in a.items() if k not in SKIP and v]
        if tag not in VOID:
            self.stack.append((tag, name))

    def handle_endtag(self, tag):
        for i in range(len(self.stack) - 1, -1, -1):
            if self.stack[i][0] == tag:
                del self.stack[i:]
                return

    def handle_data(self, data):
        text = " ".join(data.split())
        if text and self.stack:
            self.out.append((self.stack[-1][1] + "#text", text))

def lost_fields(card_html, record_text):
    """Values in the card whose tokens are not in the record. record_text: norm() of the whole record, space-padded."""
    p = CardFields()
    p.feed(card_html)
    return [(n, v) for n, v in p.out if norm(v) and f" {norm(v)} " not in record_text]
Code language: Python (python)

On the four test sites, this step found what the block check could not:

  • A stock state on a button. “Notify Me” in the button’s text and its title, on 9% of the bookstore’s cards.
  • A seller id inside a click handler. Every card’s button carried a call like AddtoCart('48213907', 1, '355', 1). The first argument was the product id the parser already kept. Rather than guess the rest from its shape, read the site’s own listing script: it defined the function as AddtoCart(productId, quantity, vendorId, productTypeId). So 355 named a seller, and the crawl held 61 different ones.
  • Image alt text carrying the brand. On a marketplace, 4,484 cards had alt text the record did not hold, often with the brand name the visible title left out.
  • Color swatch links to sibling products. A few cards linked to other product ids of the same family through data attributes: 33 links on 11 cards, exactly the input a “group these variants” feature needs.

Name a value from the site’s own code, not from its shape; a guessed field name is a guess stored forever.

Keep generic “chrome” rules away from short values

Most audits start with a default rule that buttons, forms and labels are page furniture. For a page’s blocks that rule is right. Applied to a card’s fields, it hid the stock state, because a card’s button is precisely where a store puts “Notify Me”. Register the exact value you mean to ignore, with a pattern on the value itself: the label Add to Cart only, so that Notify Me on the same attribute still fails.

Step 4: Account for the page outside the cards

Remove the card markup from the page, split what remains into blocks, and check them against every card record on the page together. Expect filters with counts, the category menu, a result total, the category’s SEO description and, on large stores, other products shown outside the result grid: sponsored banners and “related to your search” strips. Each one is a decision: register it with its reason, or treat it as something the crawler should collect.

Text that appears identically on a large share of the pages, here 30% of them and at least ten, is the site’s frame (header, footer, pop-ups) and is set aside automatically. Watch that threshold; it caused one of the false positives below.

Step 5: Compare the stored record with a re-parse of the same run

The first four steps test today’s parser. This one tests the data you already stored. For each stored record, re-parse the page from the run that wrote it and compare the two. Use that same run, not the newest page: a record a later crawl touched holds that later visit’s values, and comparing it with an older page reports a difference that is not there.

On the bookstore, 2,045 of 2,058 stored records matched their re-parse exactly. The other 13 were the store’s own duplicates: the same ISBN listed two or three times on one page under consecutive internal ids, same title, price and seller. The record is keyed by ISBN, so they collapsed into one. That is not a parser fault, and the right response is to detect it and report it, never to rewrite the store’s data to look tidy. On one marketplace, 366 of 5,891 records differed only in URL tracking parameters.

Write a register of what you leave out on purpose

Not every value belongs in the record: a lazy-load placeholder image, a broken-image script, a “No Review Yet” label that the record already expresses as a missing rating. Each of those goes into a register, one entry per decision, with the reason written down:

{"entries": [
  {"signature": "^field:img\\.book-cover@src$", "text": "lazy-loading\\.gif$",
   "kind": "not-content", "reason": "Lazy-load placeholder. The real cover is in data-src, which is kept."},
  {"signature": "^field:button\\.cart-btn@title$", "text": "^Add to Cart$",
   "kind": "not-content", "reason": "The label of a card that can be bought. Only this label: 'Notify Me' is a stock state."}
]}
Code language: JSON / JSON with Comments (json)

Three rules keep the register honest. Add an entry only after reading the blocks it covers. Never register a real product fact the parser happens to miss: it stays a finding until the parser keeps it. And keep entries in step with the parser, because an entry that matches more than you meant hides a drop.

The gate is one line of logic: exit 1 when cards, fields or the page hold any loss that is neither in the record nor in the register. Run it after every parser change. A decision you deferred (“not collecting that marker for now”) stays unregistered on purpose, so the gate keeps reminding you it is open.

A register entry says “left out on purpose, because…”; a missing entry says nobody decided.

The audit’s own false positives

The first runs flagged values that had in fact reached the record. Each came from representation, and each was fixed by making the comparison canonical, never by loosening the match:

  • Thousands commas. The page printed 1,554; the record held 1554. Join a comma between two digits before comparing.
  • 4.0 against 4. The record stored a rating as the float 4.0; the page printed 4. Render a whole-number float both ways.
  • Text printed twice. A card printed the same text twice, and the record kept it once. Register the repeat with that reason.
  • A value spread over two text nodes. A record that keeps a card’s text node by node holds “FREE delivery” and “Thu, 12 Nov” as two strings. Also check each list of strings joined in its order.
  • A site frame that fell below the template share. Once the crawl went deep, most saved pages were “load more” replies carrying cards and no header or footer. The frame then appeared on under 30% of pages and surfaced as hundreds of findings. Register the frame explicitly, with that reason.

Loosening the match (“80% of the words is close enough”) would have silenced all five, and every real drop with them.

What the audit found on four stores

SitePages / cardsFirst runAfter the parser fix
Bookstore156 / 3,007Awards, stock button, seller id, one unexplained badge275 pages / 5,387 cards: no findings
Marketplace A210 / 6,766Image alt with brand, swatch links, a storefront marker, one customizable productAlt and swatches kept; the marker deferred, so the gate still fails on it
Marketplace B208 / 7,932Price, list price and discount glued into one string; no product image; a badgeText kept node by node, images kept
Electronics retailer652 / 13,557No findingsNothing to fix

The electronics retailer came out clean for a simple reason: its listing data arrives as a JSON object per product, and the parser stores that object whole. When the source is already structured, keeping it verbatim beats picking fields out of it.

Marketplace B’s finding was a different kind of loss: the parser kept the text, but the browser had glued the values together before the parser ever saw them, which is how innerText reads inline elements.

The short version

Tests check what a scraper parser returns, so they cannot see what it skipped. Save the raw pages, re-parse them with the current parser, and account for every block, every short value and every attribute in each card, the page outside the cards, and the stored records against a same-run re-parse. Write what you leave out on purpose into a register with reasons, fix the comparison when it raises false positives, and gate on anything unregistered.

Frequently Asked Questions

Will the audit catch a value the parser reads wrongly?

No. It checks that a value from the page reached the record, not that it landed under the right name. A seller id stored as the price would pass. Pair the audit with a few field-level tests that check known cards’ values in their proper fields.

How often should I run it?

After every parser change, as a gate, and from time to time on freshly saved pages. Sites add fields and move values without notice, and a new field shows up as a new finding on the next run. A store you automate without owning it can change under you at any time.

What about pages built by JavaScript?

Audit what the browser rendered, saved as HTML, rather than your crawler’s own extract of it. If the crawler saves a JSON of selected values, that file is already a parse, so keep the rendered page beside it and audit the cards from the page.

Share post:

Subscribe

Popular

More like this
Related

Can AI CLI Conversation History Leak Answers Into a Blind Test?

You use an AI command-line tool to grade or answer hundreds of questions, one fresh run each. Then you find out it keeps every conversation on disk, answer keys included. Is run 200 quietly reading run 199?

The Code Snippets 403 That Comes From Your Host’s Firewall, Not WordPress

You change a few lines in a snippet, press Save, and get "Could not update snippet. Request failed with status code 403". Press Save on the unchanged snippet and it works. Your login is fine. Something between you and WordPress is reading the code itself.

Why a Flaky Test Passes Alone and Fails When the Machine Is Busy

Run the test by itself and it passes, every time. Run the whole suite in parallel and it fails, but only some days, and its failure looks exactly like a real bug. In our case the code was fine. The test was waiting on a clock.

Why Multisite Subsite Registration Sends Visitors to the Main Site

You put a "create an account" link on one site of a WordPress multisite. Visitors who click it land on the main site, which either says "Registration has been disabled" or signs them up there instead.