The failure is quiet, which is the problem
When you `curl` a modern page and parse the result, you usually do not get an error. You get a document. It has a `<title>`, it has some markup, it returns HTTP 200. Your parser runs happily and finds nothing, and the nothing looks like an empty result rather than a broken one.
That distinction matters enormously if you are building a pipeline. "This page has no data" and "I could not read this page" require completely different responses, and a 200 with a shell of HTML collapses them into one.
A measurement
We ran the same URL — the National Funeral Directors Association's media page — two ways on 21 September 2026.
Rendered in a real browser, with JavaScript allowed to execute:
``` status : 200 title : NFDA Media Center | Press Resources & News text : 4,338 characters links : 100 ```
The page's navigation, its body copy, and every one of those hundred links are assembled client side. A fetch that stops at the server's response does not see them, because at that moment they do not exist yet.
This is not a trick page. It is an ordinary trade-association site.
What "rendering" actually costs
Running a browser is more expensive than running a fetch, and it is worth knowing by how much rather than guessing.
Measured across four sites on the same machine:
| Page | Render time | |---|---| | example.com | 1.3s | | news.ycombinator.com | 0.8s | | nfda.org media page | 2.1s | | en.wikipedia.org article | ~1.5s |
So: roughly one to two seconds, against perhaps a tenth of that for a plain fetch. Memory is the sharper cost — a headless Chromium instance is measured in hundreds of megabytes, not kilobytes, which is why concurrency is the thing that bites first if you run this yourself.
When you do not need this
Plenty of the web still works fine with a fetch, and you should use one when it does:
- Real RSS and Atom feeds. XML is served whole. - JSON APIs. If a site has one, it is almost always better than scraping its HTML. - Server-rendered pages. Many documentation sites, government sites and older publications still send you the complete document.
The test is simple: fetch the page, search the raw response for a string you can see in your own browser. If it is absent, the content is being built client side.
When you do
- The content you can see is missing from `curl` output. - The page is a single-page app — a near-empty `<body>` with a large JavaScript bundle. - Links are generated from data rather than written into the markup. - You are feeding an AI agent that will otherwise confidently summarise an empty page.
That last one is underrated. A language model handed a shell of HTML does not error. It produces a plausible summary of nothing, and you have no easy way to tell.
What we built
The Kcalbin Browser Automation API renders a public page in a real headless browser and returns the title, the visible text and every link, over MCP — so it works inside Claude and other MCP clients with nothing to install locally.
What it does not do, stated plainly because it matters before you rely on it: no proxy rotation, no residential IPs, no anti-bot bypass, no geotargeting. It is read-only — it renders and extracts, it does not click, fill forms or log in. Private, internal and cloud-metadata addresses are refused outright, and so is any hostname that resolves to one.
If you need rotating residential proxies or millions of renders a month, buy a dedicated scraping platform. They are good at that and this is not trying to be one.
Free preview: the real title and the first 500 characters of a real render, no account. Enough to check it works on your pages before paying anything.