Why innerText Glues Your Scraped Values Together

Date:

Your scraper reads a product card and stores ₹1,249₹1,39910% off. Three values went in and one string came out, and no code can reliably split it back.

The short answer: innerText returns an element’s text as the page renders it, and elements that sit side by side in a line (inline or inline-block) get no separator when the markup has no whitespace between them, however far apart CSS draws them. So a price, a list price and a discount (₹1,249, ₹1,399, 10% off) come out as ₹1,249₹1,39910% off, and an average score with its count (4.5, 720 Ratings) comes out as 4.5720 Ratings. Keep the original string if your code already reads it, and collect the card’s text node by node with a TreeWalker beside it, plus the card’s images. Each text node is one value, so nothing has to be split.

The symptom: numbers that run into each other

A crawler drives a browser through a marketplace’s search results and keeps each product card’s text with Playwright’s inner_text(), which reads the element’s innerText. Titles look fine. The price line does not:

₹1,249₹1,39910% off
4.5720 RatingsCode language: CSS (css)

Try to read it back. ₹1,39910% off could be ₹1,399 and 10% off, or ₹13,991 and 0% off. 4.5720 could be an average of 4.5 over 720 ratings, or 4.57 over 20. A regular expression can guess from typical ranges, and every guess is a value you did not observe. In the affected crawl, price, list price and discount ran together on about 7,850 of 7,932 cards, and rating and rating count on 325. Only the first price could be read back with confidence.

A glued string has already thrown away the boundary; no parser downstream can put back what the browser removed.

Step 1: Reproduce it on a blank page

Before changing the crawler, pin down which layouts glue. The test below builds the same card in six ways and reads it three ways in headless Chromium:

from playwright.sync_api import sync_playwright

CARD = ('<div class="price"><X>₹1,249</X><X>₹1,399</X><X>10% off</X></div>'
        '<div class="rate"><X>4.5</X><X>720 Ratings</X></div>')
WALK = """(root) => {
  const out = []; const w = document.createTreeWalker(root, NodeFilter.SHOW_TEXT); let n;
  while ((n = w.nextNode())) { const t = n.nodeValue.replace(/\\s+/g, ' ').trim(); if (t) out.push(t); }
  return out; }"""

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    html = CARD.replace("<X>", "<span>").replace("</X>", "</span>")
    page.set_content(f"<div id=card>{html}</div>")
    card = page.locator("#card")
    print(repr(card.inner_text()), repr(card.text_content()), card.evaluate(WALK))
Code language: Python (python)

Swapping the <span> for other elements and layouts gave this (Chromium 145):

How the values are laid outinnerTexttextContentText nodes
<span>s, markup with no spaces₹1,249₹1,39910% offgluedseparate
<div>s set to inline-block₹1,249₹1,39910% offgluedseparate
<span>s with a CSS margin between them₹1,249₹1,39910% offgluedseparate
List price in an <s> strike-through₹1,249₹1,39910% offgluedseparate
<span>s with a newline between them in the markup₹1,249 ₹1,399 10% offkeeps the markup’s newlineseparate
<div>s as items of a flex or grid rowone value per linegluedseparate
Plain block <div>sone value per linegluedseparate

Only the text-node list kept every value apart in every layout.

Why the page looks spaced but the text is not

innerText follows the rendering rules in the HTML standard: a block-level box adds a line break before and after itself, a table cell adds a tab, and a <p> adds a blank line. Inline content is joined the way it would be read along a line of text, using only the whitespace that is actually in the markup. In these tests, flex and grid items counted as blocks and got their own lines.

A gap you see between two prices is usually a CSS margin or padding. That is layout, not a character, so innerText has nothing to emit. Sites that ship minified markup, with no whitespace between tags, therefore glue every inline neighbor. Copy the line from the page with a mouse and you get the same run-together string.

textContent is no escape. It ignores CSS entirely and returns the text nodes joined with nothing between them, so it glued even the block layouts. MDN’s page on innerText notes one more case: an element that is not rendered (for example display: none) returns the same as textContent. In the test, two block <div>s inside a hidden parent came back as Price 1249MRP 1399, with no line break at all.

Margins draw a gap on screen, but they put no character between the values for innerText to keep.

Step 2: Read the card text node by node

The DOM already holds the values apart: each price is its own text node. A TreeWalker visits text nodes in document order, so collect them as a list instead of asking for one rendered string. Collect the card’s images while you are there; a listing card’s first image is usually the product photo and the rest are badges.

(el) => {
  const root = el.closest('[data-card]') || el;            // the whole card, not just the link
  const nodes = [];
  const walker = document.createTreeWalker(root, NodeFilter.SHOW_TEXT);
  let n;
  while ((n = walker.nextNode())) {
    if (n.parentElement && n.parentElement.closest('script,style,noscript')) continue;
    const t = (n.nodeValue || '').replace(/\s+/g, ' ').trim();
    if (t) nodes.push(t);
  }
  const imgs = [];
  for (const img of root.querySelectorAll('img')) {
    const src = img.getAttribute('src') || '';
    if (src && !src.startsWith('data:') && !imgs.includes(src)) imgs.push(src);  // skip inline placeholders
  }
  return { nodes, imgs };
}
Code language: JavaScript (javascript)

Pass it to locator.evaluate() in Playwright, or run it in any page context. The price line now arrives as ['₹1,249', '₹1,399', '10% off'] and the rating as ['4.5', '720 Ratings'].

A value the site splits across elements, such as <span>₹</span><span>1,249</span>, comes back as two nodes. That is fine: the list is lossless and in order, so joining a currency sign to the number after it is a small, safe step downstream. Joining is always possible later; splitting a glued string never is.

Step 3: Keep the original string beside the new list

If extraction code already reads the innerText string (pulling a title out of its first line, say), do not replace it. Store text_nodes and card_images as new fields next to it. Adding a field changes nothing for existing readers; changing one they parse is a migration. Move each reader to the node list when you next touch it, and drop the old string only when nothing reads it.

The same trap in static HTML parsers

Parsing saved HTML without a browser has the same failure under other names. In BeautifulSoup, get_text() with no argument returned ₹1,249₹1,39910% off on the same markup, while get_text(" | ") returned ₹1,249 | ₹1,399 | 10% off and stripped_strings gave the three values as a list. lxml’s text_content() behaves like textContent. Always pass a separator or iterate the strings.

Check the fix on saved pages

A browser crawl usually saves the raw page next to what it extracted. Run the new script offline against a saved page in a local Chromium and compare: on the test crawl’s saved pages it returned '₹1,249', '₹1,399', '10% off' as separate values. Across 208 saved pages and 7,932 cards, price, list price, discount, rating and product image were readable on every card. Records stored before the fix can be filled the same way, from their saved pages, without fetching anything again. To find what else a card shows that the record never keeps, audit what the scraper parser drops against those same saved pages.

Two panels comparing how many of a product card's five values (price, list price, discount, rating and rating count) innerText, textContent and a TreeWalker over text nodes read back apart, measured in headless Chromium. Panel one, values in inline elements with no whitespace between them: innerText and textContent each get 1 of 5, both flagged red because the values run together, while the text nodes get all 5. Panel two, the same values as items of a flex row: innerText gets all 5 because flex items get their own lines, textContent still gets only 1, flagged red, and the text nodes get all 5.

The short version

innerText gives you text as rendered, and inline neighbors with no whitespace between them in the markup render as one run, whatever spacing CSS draws. Prices, list prices, discounts and rating counts are exactly that shape on product cards. Read the card’s text nodes with a TreeWalker and keep its images, store them beside any string your code already reads, and in static parsers always pass a separator.

Frequently Asked Questions

Can I use Playwright’s all_inner_texts() instead?

It returns one innerText per element a locator matches, so it separates values when you can target each value’s own element. On sites with generated class names those selectors break often. Walking a card’s text nodes needs only a selector for the card itself.

Why not split the glued string with a regular expression?

Because the boundary is ambiguous. ₹1,39910% off reads correctly as ₹1,399 and 10%, but nothing in the string rules out ₹13,991 and 0%. Any split rule is a guess about the data, and the wrong guesses look exactly like right ones.

Does innerText work on hidden cards?

Not as you would expect. An element that is not rendered, such as a collapsed panel with display: none, returns its textContent, so even block-level values run together. Text nodes are read the same way whether the card is visible or not.

Share post:

Subscribe

Popular

More like this
Related

Can AI CLI Conversation History Leak Answers Into a Blind Test?

You use an AI command-line tool to grade or answer hundreds of questions, one fresh run each. Then you find out it keeps every conversation on disk, answer keys included. Is run 200 quietly reading run 199?

The Code Snippets 403 That Comes From Your Host’s Firewall, Not WordPress

You change a few lines in a snippet, press Save, and get "Could not update snippet. Request failed with status code 403". Press Save on the unchanged snippet and it works. Your login is fine. Something between you and WordPress is reading the code itself.

Why a Flaky Test Passes Alone and Fails When the Machine Is Busy

Run the test by itself and it passes, every time. Run the whole suite in parallel and it fails, but only some days, and its failure looks exactly like a real bug. In our case the code was fine. The test was waiting on a clock.

Why Multisite Subsite Registration Sends Visitors to the Main Site

You put a "create an account" link on one site of a WordPress multisite. Visitors who click it land on the main site, which either says "Registration has been disabled" or signs them up there instead.