How we measure
Real questions agents asked, an answer key checked on the primary source, deterministic grading, and a date on every number.
Real questions, not a benchmark set
The search eval starts from questions AI agents actually sent to search tools while doing real software and research work. We wrote 48 questions from them, eight in each of six classes: dev, docs, news, policy, price and spec. Personal projects were excluded, and the full question list isn't published; six samples are.
An answer key checked on the primary source
For every question we found the primary source (the vendor's docs, pricing page, changelog or the regulation itself) and quoted the passage that answers it. A script then re-fetched every source page and string-matched the quote: 28 matched exactly, 6 matched closely, and in 10 every required fact was on the page. We checked one datasheet by hand. Three questions were dropped (one had no primary source, one had its only source in an image, one had its source page change), which leaves 45 graded questions.
Deterministic grading
No model judges the search results. Each question lists the facts a correct answer must contain, and grading is a string match:
- fact@5: the share of required facts present in the text an API returned for its top five results.
- Full answers: questions where every required fact appeared.
- Answer page first and in top 3 pages: the same check after reading the top one or three result pages in full, the way an agent searches and then fetches.
- A call that fails scores zero on every measure, because it fails the agent too. Rows with failures say how many questions they answered.
- Latency and context size are medians over the calls that returned.
Answer APIs return a paragraph rather than results. On the answer page, 5 of them got the same 45 questions; every answer was string-matched first, and every miss was read by hand against the key and judged correct, partly right, wrong or declined.
Fetch and extraction
The fetch page sends one call per URL to each tool on two sets: the 45 primary pages from the search answer key, graded on the same required facts, and 15 pages that block bots (review and job sites, social apps, company profiles, large PDFs). For the blocked pages we read two truth phrases from a real browser render the same evening; a page counts as read when both are in the returned text. Measured Nov 8, 2026.
One run, dated
Each API ran once, on Nov 7, 2026, within minutes of the answer key being checked, so every API saw the same live web. Search results change daily; a number without a date isn't comparable with anything.
Evidence badges
We ran it ourselves with the method on this page. Today that is 4 categories: web search, answer APIs, fetch and extraction, and Context7 and DeepWiki on the dev and docs questions.
A third party published it. We cite the publisher and the date beside every figure, note who runs the board, and record where boards disagree.
Published by the vendor itself, or by a study a vendor paid for. Shown only beside independent figures, and never used to order a list.
Desk research from vendor pages, GitHub and secondary sources, compiled on a stated date; prices are not individually re-verified. Ordered by adoption (stars) where one metric applies to the list, otherwise alphabetically. Never shown as a score.
Re-measurement
Rankings in this field go stale within months: one code-review board had four different leaders in 2026. We re-run our measurements when a tool ships a major change and at least quarterly, show the date of every run, and keep earlier results rather than overwriting them. You.com was tried but not measured yet: the account's free credit read as depleted. Docs tools (Context7, DeepWiki) were run on the 15 dev and docs questions; see the docs page.