cloudflare just dropped a /crawl endpoint and everyone's losing their mind.

calm down. let me tell you what it actually is, what it isn't, and why you probably don't need it.
the /crawl endpoint is a wrapper. you give it a URL, it spins up headless browsers on Cloudflare's infra, follows links, renders JavaScript, and hands you back markdown or JSON. all with 1 API call.
it's cool but not revolutionary.
firecrawl does this. crawl4AI does this. spider does this. they've been doing it for months. cloudflare just added it to their existing Browser Rendering product and everyone acted like they invented crawling.
what IS interesting: it's cloudflare. that means it's cheap ($0.09/hour).
but the thing is you probably don't even need a crawler.
there are 8 ways an AI agent can read a webpage. most jump straight to the complex ones when a 50ms HTTP request would've done the job. so let's break them all down, from the simplest to the most overkill.
1. raw HTTP fetch
your agent sends a request, gets back HTML. that's it.
like reading a book's source code instead of the printed page. works great for simple sites, blogs, wikis, docs. breaks on anything that uses JavaScript to load content.
speed: ~50ms. cost: free.
2. readability parser
same thing, but with a cleaning step. strips out nav bars, ads, footers, cookie banners. gives you just the article text in clean markdown.
doesn't handle JavaScript-rendered content. but for articles and docs, it's perfect, and it's what I use daily.
speed: ~100ms. cost: free.
3. headless browser (local)
launches an invisible Chrome that loads the page like a human would. JavaScript runs, content renders, everything loads. you can click, scroll, fill forms, log in.
the problem: slow (2-10s), eats ~200MB RAM per instance, and you maintain the infra.
tools: Playwright, Puppeteer, Selenium.
4. cloud browser API
same as #3 but someone else runs the browser. you send a URL, get back the rendered page. this is where Cloudflare's /crawl lives, along with Browserbase and Steel.
no infra headaches, scales easily, cheap. tradeoff: less control over interactions.
5. managed scraping API
this is the anti-bot warfare tier. ScrapingBee, Bright Data, rotating proxies, CAPTCHA solving, residential IPs. for when the site actively fights you.
works. costs $49-499+/month.
6. AI-native crawler
Firecrawl, Crawl4AI, Spider. crawl + render + auto-convert to clean markdown/JSON. built for RAG pipelines. define extraction schemas in natural language.
the "new wave" that Cloudflare is now competing with.
7. LLM extraction
skip the code entirely. dump the page content into an LLM, ask "what's the price?" in plain english. no CSS selectors, no regex, no maintenance when the site redesigns.
downside: expensive at scale (tokens add up fast). best as the final step after cleaning with methods 1-6.
8. official APIs
the one everyone forgets. X, Reddit, most SaaS, they have APIs. structured data, no parsing, no anti-bot games. when an API exists, it's always the right choice.
the good setups combine 2-3:
→ fetch → readability → LLM for cheap article extraction
→ cloud browser → LLM for JavaScript-heavy sites
→ sniff the actual API in DevTools → call it directly, the holy grail, free, fastest, most reliable
→ AI crawler → vector DB for full know ledge bases
real costs at 10,000 pages/month
• HTTP Fetch: $0
• Jina Reader: $0
• Cloudflare Browser: ~$5
• Spider: ~$4.80
• Firecrawl: $47/mo
• ScrapingBee: $49-147/mo
• Bright Data: $499+/mo
2 rules i follow:
start simple. API > fetch > readability > browser. only add complexity when the simpler method fails. I see people spinning up Playwright for sites where curl works fine.
most sites don't need JS rendering. 60%+ of the web is static or server-rendered. test with a simple fetch first.