Fetch, scrape and extraction

We fetched 15 pages that turn bots away (G2, Glassdoor, LinkedIn, Yelp and others) and 45 ordinary primary pages with 10 fetch tools and settings. On ordinary pages they are nearly level. On blocked pages they split widely, and no tool read more than 10 of the 15.

Measured by us Category 3 of 11

Pages that block bots, ranked

15 pages agents hit that turn bots away: review and job sites, company profiles, social apps, two large PDF datasheets, a shop and a Q&A site. Ranked by full pages read (both truth phrases in the returned text), then by the share of phrases found. Equal full pages and an equal phrase share to two decimals share a rank. Hover or tap a dotted value for its source.

Measured by usOur fetch eval, one run · Nov 8, 2026
Fetch tools ranked on 15 pages that block bots, by full pages read, then share of truth phrases found. Measured Nov 8, 2026.
Rank Fetch tool Full pages Phrases found Calls OK Median chars Median time Cost / 1k pages
1 Exa contents with live-fetch first live GET, then Exa's stored copy · a two-step recipe, not one product 10/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text Nov 8, 2026 0.73 15/15 12k 1.9 s Free, then $1 The live GET is free; Exa contents is $1 per 1,000 pages when it falls back to the stored copy
2 Parallel Extract full content 9/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text Nov 8, 2026 0.80 15/15 53k 0.25 s $1
=3 Exa Contents /contents 8/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text. Tied: same full pages and 0.60 of truth phrases Nov 8, 2026 0.60 12/15 14k 0.23 s $1
=3 Tavily Extract advanced and basic, identical results 8/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text. Tied: same full pages and 0.60 of truth phrases Nov 8, 2026 0.60 10/15 47k 0.31 s 1 cr / 5 URLs 1 credit per 5 URLs
5 Firecrawl /v2/scrape 7/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text Nov 8, 2026 0.60 11/15 53k 0.61 s 1 cr / page Free tier used: 1 credit per page; a PDF bills 1 credit per PDF page (182 credits for the two datasheets)
6 Jina Reader r.jina.ai 7/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text Nov 8, 2026 0.53 13/15 15k 8.3 s Token-billed Billed by tokens returned; measured on free trial tokens
7 Linkup fetch renderJs 6/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text Nov 8, 2026 0.40 7/15 23k 10.5 s $5
8 TinyFish Fetch /fetch 5/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text Nov 8, 2026 0.43 11/15 6.8k 4.3 s Free Free, 1,000 fetches a day
9 Plain HTTP GET browser User-Agent; PDFs via pdftotext 3/15 Our fetch eval, 15 pages that block bots Pages where every truth phrase appeared in the returned text Nov 8, 2026 0.20 4/15 18k 0.18 s Free
  • No tool read every page: 14 of the 15 were fully read by at least one tool, but none read more than 10. Nobody returned the Indeed company page with both phrases.
  • Exa contents with live-fetch first leads, and it is a two-step recipe rather than one product: a live GET first, then Exa's stored copy when the site turns the GET away. Stored copies get past bot walls. The best single API is Parallel Extract: 9/15 pages, and the highest share of phrases found (0.80).
  • Only Firecrawl read the Glassdoor company page and only Parallel Extract read the Yelp business page.
  • Stored copies miss current text on pages that change: on the Glassdoor company page, the Amazon product page and the Investing.com quote page, Exa Contents's stored copy came back without the phrases a live fetch found.
  • Firecrawl refuses some sites: it answered "we do not support this site" for the TikTok profile and the Yelp business page, and the LinkedIn company page needs a paid add-on.
  • Large PDFs are solved by all: every API returned both datasheets with both phrases, except TinyFish Fetch on the 2 MB ST datasheet (text, but not both phrases). Plain HTTP GET timed out on the 2 MB ST datasheet.

Page by page

The 15 blocked pages, by site kind, with the protection each used on the day. Columns follow the ranking.

Measured by us
PageExa + liveParallelExaTavilyFirecrawlJinaLinkupTinyFishHTTP
G2 product reviewsDataDome ●◐●×●××××
Glassdoor company pageCloudflare Turnstile ◐◐◐×●○×××
Indeed company pageCloudflare ○◐○◐◐○×○×
Crunchbase profileCloudflare challenge ●◐●×◐○×◐×
Similarweb site profileJS-rendered metrics ●●●××○×××
X profileJS-only app ●●×●◐◐●○●
TikTok profileJS-only app ○●×●×●●◐○
Large PDF datasheetST, 2 MB ●●●●●●●◐×
Large PDF datasheetTI, 4 MB ●●●●●●●●●
Amazon product pagebot interstitial ●●○●●●×●●
Yelp business pageDataDome ○●×××××××
Medium articleCloudflare ●●●●●●●●×
LinkedIn company pageauth wall ●●●●×●○●×
Investing.com quote pageCloudflare ◐◐◐●●●●○×
Stack Overflow questionCloudflare ●◐●◐◐◐×●×
Full pages1098877653

Scroll the table sideways for all 9 tools.

  • ●both phrases
  • ◐one of two phrases
  • ○text came back without the phrases
  • ×the call failed or was refused

Ordinary docs pages: all close

45 primary pages from the search answer key: docs, pricing, changelogs, policies, specs and news. Every tool read between 33 and 37 of the 45 in full, a spread of 4 pages on one run, so this set barely separates them. Shown in the same order as above (full pages, then facts found), without ranks. 37 pages were fully read by some tool; on the other 8, the required facts span more of the page than any extraction kept.

Measured by usOur fetch eval, one run · Nov 8, 2026
Fetch tools on 45 ordinary primary pages, in the same order, not ranked. Measured Nov 8, 2026.
Fetch tool Full pages Facts found Calls OK Median chars Median time Cost / 1k pages
Firecrawl /v2/scrape 37/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.93 45/45 16k 0.42 s 1 cr / page Free tier used: 1 credit per page; a PDF bills 1 credit per PDF page (182 credits for the two datasheets)
Exa contents with live-fetch first live GET, then Exa's stored copy · a two-step recipe, not one product 36/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.92 45/45 13k 1.4 s Free, then $1 The live GET is free; Exa contents is $1 per 1,000 pages when it falls back to the stored copy
Linkup fetch renderJs 36/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.91 45/45 20k 1.5 s $5
Exa Contents /contents 36/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.90 44/45 11k 0.23 s $1
Jina Reader r.jina.ai 35/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.90 45/45 15k 3.8 s Token-billed Billed by tokens returned; measured on free trial tokens
TinyFish Fetch /fetch 34/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.91 45/45 9.9k 1.0 s Free Free, 1,000 fetches a day
Tavily Extract advanced 34/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.90 45/45 14k 0.23 s 2 cr / 5 URLs 2 credits per 5 URLs
Tavily Extract basic · other setting, not ranked 33/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.87 44/45 13k 0.24 s 1 cr / 5 URLs 1 credit per 5 URLs
Parallel Extract full content 34/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.88 44/45 29k 0.23 s $1
Plain HTTP GET browser User-Agent; PDFs via pdftotext 34/45 Our fetch eval, 45 ordinary primary pages Pages where every required fact appeared in the returned text Nov 8, 2026 0.85 41/45 18k 0.39 s Free

Tried, not measured: You.com Contents. The account's free credit read as depleted.

How the fetch eval works

  • One call per URL, one run, each tool at its own settings beyond the one named. A failed or refused call scores zero.
  • Blocked pages: two truth phrases per page, read from a real browser render the same evening. "Full" means both appear in the returned text; "phrases found" is the share across all pages.
  • Ordinary pages: the required facts from the search answer key, checked the same way as search results.
  • Median characters are over calls that returned text. Median time is over every call, because an agent waits for a refusal too.
  • Tavily Extract basic and advanced returned identical results on the blocked pages, so they share one row there.
  • Plain HTTP GET is the baseline: one request with a browser User-Agent, PDFs converted with pdftotext.
Independent benchmarkopenbenchmarks.com, web-search-for-coding-agents · snapshot Sep 15, 2026

Independent: coding tickets completed, search and fetch together

  1. Exadeep 83.0%23.7k tok
  2. Exaauto 81.7%27.4k tok
  3. TinyFish 79.0%12.8k tok
  4. Perplexityhigh 77.7%20.1k tok
  5. Paralleladvanced 77.0%27.1k tok
  6. Firecrawl 76.0%17.4k tok
  7. Tavilyadvanced 60.0%26.3k tok

Mean of 3 runs; tokens are the agent's context per ticket. The board says to re-read the live page before quoting.

Tools

Alphabetical. Links go to each vendor's own site.

Catalog only, not yet measuredvendor pages and secondary sources · compiled Nov 7, 2026
ToolWhat it doesEvidencePrice or terms
Bright Data proxy network and scraping browser none yet plans by volume; browser render and CAPTCHA solving included
Exa Contents clean page text for URLs 8/15 blocked pages read (#=3 of 9)36/45 ordinary pages83.0% with Exa deep search (search + fetch) $1/1k
Firecrawl URL to markdown; crawl and extract 7/15 blocked pages read (#5 of 9)37/45 ordinary pages76.0% (search + fetch) 1,000 credits/mo free; keyless tier
Jina Reader URL to markdown by prefixing r.jina.ai 7/15 blocked pages read (#6 of 9)35/45 ordinary pages free without a key at 20 requests/min
Kagi Extract page content extraction API none yet $4/1k
Linkup fetch URL to text, with an optional JavaScript render 6/15 blocked pages read (#7 of 9)36/45 ordinary pages $5/1k
Oxylabs proxy network and scraper APIs none yet plans by volume
Parallel Extract page content extraction API 9/15 blocked pages read (#2 of 9)34/45 ordinary pages77.0% with Parallel advanced search (search + fetch) $1/1k
ScrapingBee scraping API with headless browsers none yet plans by volume
Tavily Extract page content extraction API 8/15 blocked pages read (#=3 of 9)34/45 ordinary pages60.0% with Tavily advanced search (search + fetch) ~$1.60/1k
TinyFish Fetch search and fetch, free 5/15 blocked pages read (#8 of 9)34/45 ordinary pages79.0% (search + fetch) free
You.com Contents page content extraction API none yet $1/1k
ZenRows scraping API with anti-bot bypass none yet plans by volume

Caveats

  • One run per tool. Truth phrases on the blocked pages were read once from a real browser render the same evening; pages that change (newest reviews, headlines) penalise a tool that returns a slightly older copy.
  • The independent chart above scores search and fetch together on coding tickets; it is a different task from ours and ranks the tools differently. On the same site's search-only board Perplexity leads at 77.3% and Firecrawl is second at 70.3%.

Other categories