Headless Browser

A headless browser is a genuine web browser running with no user interface. The word headless simply means the "head" — the window, tabs, and buttons you normally see — is missing. Everything else is intact: it loads pages, executes JavaScript, applies CSS, stores cookies, and renders layout, all invisibly in the background.

It is the same engine as the browser on your desktop, just controlled by a script rather than a person.

Why not just fetch the page?

This is the key thing to understand. A simple HTTP request — curl, Python requests — downloads the raw HTML the server sends and stops there. That was fine when websites shipped finished pages.

Modern sites often send a nearly empty shell plus a bundle of JavaScript, then build the content in the browser. Fetch such a page with curl and you get a skeleton with no prices, no listings, no text. A headless browser runs that JavaScript, so it sees the finished page a real visitor would see.

The trade-off is cost. A headless browser uses far more memory and CPU than a plain request and is much slower — often hundreds of milliseconds versus a few. Sensible projects use plain requests wherever possible and reach for a browser only when the page genuinely requires one.

What it can do

• Click buttons, fill forms, and log in
• Scroll to trigger lazy-loaded or infinite-scroll content
• Wait for specific elements to appear before reading them
• Take screenshots and generate PDFs
• Intercept network requests, and block images or ads to run faster
• Run many isolated sessions in parallel, each with its own cookies

The common tools

• Playwright — Microsoft's library, driving Chromium, Firefox, and WebKit from one API. Currently the usual default for new projects.
• Puppeteer — Google's library, focused on Chrome and Chromium. Mature and widely documented.
• Selenium — the long-standing standard, with the broadest language support and deep roots in QA testing.

Being detected — and why it happens

Websites can often tell a headless browser from a human one, because automation leaves traces. The navigator.webdriver property is set to true, the user-agent may contain the word HeadlessChrome, plugin and font lists come back unusually sparse, and mouse movement is either absent or suspiciously linear. Anti-bot systems check exactly these signals.

That is why people pair headless browsers with stealth plugins, realistic headers, human-like timing, and high-trust residential or mobile proxies. Even then, the browser itself is only part of the picture — the IP address it connects from matters just as much, since a perfect browser fingerprint arriving from a flagged datacenter IP still gets blocked.

Headless is not only for scraping

Its original purpose was automated testing, and that is still its biggest use. Continuous integration pipelines run thousands of headless browser tests on every commit, because a server has no screen to display a window on. The same technology also generates PDF invoices, renders social preview images, and produces page screenshots on demand.