How automatic cookie scanning works (and its limits)

Every compliant cookie banner needs an accurate list of what the site actually stores on a visitor's device — the cookie declaration. Maintaining that by hand is hopeless: third-party tags set cookies you never named, and they change without telling you. Automatic scanning solves the discovery problem by loading your site the way a browser does and recording everything it stores. It is genuinely useful, and it has real limits worth understanding.

A scanner is, at heart, a headless browser. It launches a real Chromium instance under automation, navigates to a page, lets scripts run, and observes what happens. Because it is a real browser and not a simple HTTP fetch, it executes JavaScript, loads third-party tags, and triggers the same cookie-setting behaviour a visitor would — which is the only way to see cookies that are written by script rather than sent in a header.

There are several channels a scanner watches. Cookies arrive either in HTTP response headers or via document.cookie in JavaScript, and a headless browser can read the full cookie store after the page settles. Beyond cookies, modern tracking also uses localStorage, sessionStorage and IndexedDB, so a thorough scanner inventories those too — a value in localStorage can identify a user just as effectively as a cookie, and regulators treat it the same way.

One page is not a site. The scanner crawls: starting from a seed URL it follows same-origin links to build a representative sample of templates — product pages, articles, checkout, contact forms — because different pages load different tags. A blog post may carry a social embed the homepage does not; a checkout page may add a payment or fraud-detection cookie. Sampling breadth is a trade-off between coverage and scan time, so crawlers cap how many pages they visit per run.

Discovery is only half the job; the cookies then have to be classified. Each cookie found is matched against a database of known cookies keyed by name and domain, which supplies a category — necessary, preferences, statistics or marketing — a provider, a purpose and a typical lifespan. A well-known cookie like _ga is identified instantly as Google Analytics statistics; unknown cookies are flagged for a human to review, because a confident guess would be worse than asking.

The output is the declaration you show in the banner and preference centre: a table of cookies grouped by category, with provider, purpose and expiry. Keeping that table generated from real scans rather than typed by hand is what keeps it honest as your site changes, and it is what regulators and auditors expect to see when they look under the hood.

Now the limits, because they matter. A scanner can only record what fires while it is looking. If your site does prior blocking correctly, the marketing and analytics tags are inert until consent — so a naive scan sees nothing to catalogue. A good scanner works around this by simulating a visitor who accepts: it scrolls the page and clicks the accept button, then waits a moment so the very tags you need to catalogue are allowed to load and set their cookies.

Consent-gated behaviour is not the only blind spot. A scanner typically sees your site as an anonymous, logged-out visitor, so cookies that appear only behind a login, in a user's account area, or during an authenticated transaction can be missed. Anything gated by a specific action the crawler does not perform — completing a purchase, playing a video, submitting a form — may never surface in the results.

Third-party behaviour is also variable. Ad and tag-manager vendors change what they set over time and sometimes vary it by visitor geography, device or A/B bucket, so a single scan from one location is a snapshot, not a guarantee. A cookie that did not fire during last night's scan can still reach a real user tomorrow, which is why point-in-time results should never be treated as exhaustive.

Geography deserves its own note. Because tag behaviour, and even which vendors load, can depend on where the request comes from, a scan run from one region may miss cookies that only appear for visitors elsewhere. If you serve materially different tags by region, meaningful coverage means scanning from more than one vantage point, not assuming one is representative.

The honest conclusion is that scanning is necessary but not sufficient on its own. Because the picture drifts, scans have to be repeated on a schedule — Cookie Munch runs them daily — and paired with prior blocking so that undiscovered cookies are still held until consent. Blocking by category is what protects you between scans: even a cookie you have never catalogued cannot run if its category has not been consented to.

Cookie Munch runs scheduled scans with a Playwright-driven Chromium browser, crawls a same-origin sample of your pages, captures cookies along with localStorage, sessionStorage and IndexedDB, and classifies them against a curated known-cookie database into your consent categories, flagging anything unrecognised for review. The result feeds the declaration your banner shows — but it is deliberately paired with category-level prior blocking, because the safest way to handle a cookie you have not yet seen is to block it until the visitor says yes.