webmcpx benchmark: Gemini 3.8 Flash
Leaderboard
How much WebMCP saves, model by model
One agent reads, clicks and fills using page snapshots; the other calls the site's WebMCP tools. Pick a model; biggest savings first.
Ranked by success with WebMCP, then by cost per run.
Selected cases: 21 of 30 questions, where WebMCP used at most half the tokens. All results are public, or run it yourself with just bench.
Gemini 3.8 Flash (Google)
Cloudflare
“How many requests per day can my Workers handle on the Cloudflare Workers Free plan before hitting the limit?”
developers.cloudflare.com (opens Cloudflare)46×
fewer tokens
72k → 1.6k tokens
| Question | Without WebMCP: tokens | With WebMCP: tokens | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Cloudflare: How many requests per day can my Workers handle on the Cloudflare Workers Free plan before hitting the limit? | 72k | 1.6k | 100% | 100% |
| Cloudflare: How long can a key be in Cloudflare Workers KV? I want to know the maximum key size. | 41k | 1.5k | 100% | 100% |
| Cloudflare: What is the maximum size of a single value stored in Cloudflare Workers KV? | 34k | 1.4k | 100% | 100% |
| Cloudflare: I'm on the Workers Paid plan. What's the maximum size a single Cloudflare D1 database can grow to? | 29k | 1.4k | 100% | 100% |
| Cloudflare: What is the largest message I can send to a Cloudflare Queue? I need the per-message size limit. | 42k | 4.9k | 100% | 100% |
| Question | Without WebMCP: time | With WebMCP: time | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Cloudflare: How many requests per day can my Workers handle on the Cloudflare Workers Free plan before hitting the limit? | 14.4s | 5.8s | 100% | 100% |
| Cloudflare: How long can a key be in Cloudflare Workers KV? I want to know the maximum key size. | 40.3s | 6.8s | 100% | 100% |
| Cloudflare: What is the maximum size of a single value stored in Cloudflare Workers KV? | 11.0s | 5.2s | 100% | 100% |
| Cloudflare: I'm on the Workers Paid plan. What's the maximum size a single Cloudflare D1 database can grow to? | 10.7s | 6.9s | 100% | 100% |
| Cloudflare: What is the largest message I can send to a Cloudflare Queue? I need the per-message size limit. | 14.8s | 12.9s | 100% | 100% |
| Question | Without WebMCP: steps | With WebMCP: steps | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Cloudflare: How many requests per day can my Workers handle on the Cloudflare Workers Free plan before hitting the limit? | 5.7 | 2.0 | 100% | 100% |
| Cloudflare: How long can a key be in Cloudflare Workers KV? I want to know the maximum key size. | 6.0 | 2.0 | 100% | 100% |
| Cloudflare: What is the maximum size of a single value stored in Cloudflare Workers KV? | 5.3 | 2.0 | 100% | 100% |
| Cloudflare: I'm on the Workers Paid plan. What's the maximum size a single Cloudflare D1 database can grow to? | 4.7 | 2.0 | 100% | 100% |
| Cloudflare: What is the largest message I can send to a Cloudflare Queue? I need the per-message size limit. | 6.0 | 3.7 | 100% | 100% |
| Question | Without WebMCP: cost | With WebMCP: cost | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Cloudflare: How many requests per day can my Workers handle on the Cloudflare Workers Free plan before hitting the limit? | $0.03 | $0.0017 | 100% | 100% |
| Cloudflare: How long can a key be in Cloudflare Workers KV? I want to know the maximum key size. | $0.015 | $0.0017 | 100% | 100% |
| Cloudflare: What is the maximum size of a single value stored in Cloudflare Workers KV? | $0.014 | $0.0016 | 100% | 100% |
| Cloudflare: I'm on the Workers Paid plan. What's the maximum size a single Cloudflare D1 database can grow to? | $0.014 | $0.0016 | 100% | 100% |
| Cloudflare: What is the largest message I can send to a Cloudflare Queue? I need the per-message size limit. | $0.017 | $0.0044 | 100% | 100% |
The 2 WebMCP tools Cloudflare registers
as the agent saw them · 2026-11-08
| Tool | Description | Called |
|---|---|---|
search | Full-text search across Cloudflare developer documentation. Returns matching pages with titles, URLs, and content snippets. Use this to find documentation on any Cloudflare product or feature. search({"query":"Workers Free plan request limit per day"}) search({"query":"Workers KV limits maximum key size"}) search({"query":"Workers KV limits maximum value size"}) search({"query":"D1 limits maximum database size paid plan"}) search({"query":"Queues limits message size"}) | Called |
list-directories | Returns a list of all Cloudflare products available in the developer documentation, including each product's name, docs URL, short description, and category group. Use this to discover what products exist before searching or navigating. | –Not called |
- gemini-3.8-flash, reasoning medium, headless Chrome. Same prompt and step limit in both arms; 3 runs per question and arm.
- Without WebMCP: a DOM-snapshot agent that reads, clicks and fills.
- With WebMCP: only the tools the page registers (its own, or webmcpx’s on our test sites).
- Graded by facts in the answer and page state. No LLM judge.
- A live site: it ships its own WebMCP tools.
- 30 questions on 6 sites, 27 of them read-only; the biggest savings come from the sites' own search tools.
- 3 runs per question and arm: 3 of 3 solved is evidence, not proof of 100%.
- Time is the agent loop after the page loaded; cost is the model's API usage at dated list prices, cache included.
- Every failed run says why in the results (bot check, answer not from the page, step limit…).
git clone https://github.com/webmcpx/webmcpx
cd webmcpx && just setup
GEMINI_API_KEY=... just bench --model gemini-3.8-flash --reasoning-effort mediumspecification.website
“How long should a page's meta description be, according to this spec?”
specification.website (opens specification.website)13×
fewer tokens
64k → 4.9k tokens
| Question | Without WebMCP: tokens | With WebMCP: tokens | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| specification.website: How long should a page's meta description be, according to this spec? | 64k | 4.9k | 100% | 100% |
| specification.website: What minimum contrast ratio does this site's spec require for normal-size body text? | 50k | 4.8k | 100% | 100% |
| specification.website: What's the minimum tap target size this spec requires for buttons and other interactive controls? | 47k | 5.4k | 100% | 100% |
| specification.website: What max-age should I put in my HSTS header for production, per this spec? | 22k | 4.4k | 100% | 100% |
| specification.website: What INP value does this spec set as the target for good responsiveness? | 23k | 4.8k | 100% | 100% |
| Question | Without WebMCP: time | With WebMCP: time | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| specification.website: How long should a page's meta description be, according to this spec? | 26.1s | 7.3s | 100% | 100% |
| specification.website: What minimum contrast ratio does this site's spec require for normal-size body text? | 21.7s | 7.1s | 100% | 100% |
| specification.website: What's the minimum tap target size this spec requires for buttons and other interactive controls? | 24.0s | 8.1s | 100% | 100% |
| specification.website: What max-age should I put in my HSTS header for production, per this spec? | 14.1s | 7.2s | 100% | 100% |
| specification.website: What INP value does this spec set as the target for good responsiveness? | 13.9s | 6.8s | 100% | 100% |
| Question | Without WebMCP: steps | With WebMCP: steps | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| specification.website: How long should a page's meta description be, according to this spec? | 10.0 | 3.0 | 100% | 100% |
| specification.website: What minimum contrast ratio does this site's spec require for normal-size body text? | 8.0 | 3.0 | 100% | 100% |
| specification.website: What's the minimum tap target size this spec requires for buttons and other interactive controls? | 9.0 | 3.0 | 100% | 100% |
| specification.website: What max-age should I put in my HSTS header for production, per this spec? | 6.0 | 3.0 | 100% | 100% |
| specification.website: What INP value does this spec set as the target for good responsiveness? | 6.0 | 3.0 | 100% | 100% |
| Question | Without WebMCP: cost | With WebMCP: cost | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| specification.website: How long should a page's meta description be, according to this spec? | $0.025 | $0.0045 | 100% | 100% |
| specification.website: What minimum contrast ratio does this site's spec require for normal-size body text? | $0.025 | $0.0043 | 100% | 100% |
| specification.website: What's the minimum tap target size this spec requires for buttons and other interactive controls? | $0.022 | $0.0052 | 100% | 100% |
| specification.website: What max-age should I put in my HSTS header for production, per this spec? | $0.015 | $0.0042 | 100% | 100% |
| specification.website: What INP value does this spec set as the target for good responsiveness? | $0.013 | $0.0043 | 100% | 100% |
The 5 WebMCP tools specification.website registers
as the agent saw them · 2026-11-08
| Tool | Description | Called |
|---|---|---|
search_spec | Search The Website Specification for matching topics. Returns ranked results with title, status, category, canonical URL, and a one-line summary. Use this when the user asks about a topic by keyword (e.g. "CSP", "alt text", "llms.txt"). search_spec({"query":"meta description"}) search_spec({"query":"contrast ratio"}) search_spec({"query":"tap target size"}) search_spec({"query":"HSTS"}) search_spec({"query":"INP"}) | Called |
get_topic | Fetch the Markdown for one spec page by its slug, returned as lines you can page through. Use this after search_spec or list_topics to read the canonical guidance. get_topic({"slug":"meta-description"}) get_topic({"slug":"color-contrast"}) get_topic({"slug":"touch-target-size"}) get_topic({"slug":"hsts"}) get_topic({"slug":"core-web-vitals"}) | Called |
list_topics | List spec topics, optionally filtered by category and/or status. Returns title, status, category, summary, and URL per topic. Use this when the user wants the canonical list (e.g. "all required SEO topics"). Results are paged — follow page/page_size to walk the whole list. | –Not called |
open_checklist | Navigate the user to the full spec checklist page. Optionally jumps to a category section via URL hash. UI action — does not return data. | –Not called |
open_search | Open the on-page ⌘K search overlay so the user can search the spec interactively. Optionally pre-fills a query. UI action — does not return results; use search_spec for that. | –Not called |
- gemini-3.8-flash, reasoning medium, headless Chrome. Same prompt and step limit in both arms; 3 runs per question and arm.
- Without WebMCP: a DOM-snapshot agent that reads, clicks and fills.
- With WebMCP: only the tools the page registers (its own, or webmcpx’s on our test sites).
- Graded by facts in the answer and page state. No LLM judge.
- A live site: it ships its own WebMCP tools.
- 30 questions on 6 sites, 27 of them read-only; the biggest savings come from the sites' own search tools.
- 3 runs per question and arm: 3 of 3 solved is evidence, not proof of 100%.
- Time is the agent loop after the page loaded; cost is the model's API usage at dated list prices, cache included.
- Every failed run says why in the results (bot check, answer not from the page, step limit…).
git clone https://github.com/webmcpx/webmcpx
cd webmcpx && just setup
GEMINI_API_KEY=... just bench --model gemini-3.8-flash --reasoning-effort mediumNotion
“I'm paginating through results from the Notion API. What's the largest page_size I can request per call on the list endpoints?”
developers.notion.com (opens Notion)8.2×
fewer tokens
29k → 3.5k tokens
| Question | Without WebMCP: tokens | With WebMCP: tokens | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Notion: I'm paginating through results from the Notion API. What's the largest page_size I can request per call on the list endpoints? | 29k | 3.5k | 100% | 100% |
| Notion: When my Notion integration sends too many requests, what error code string does the API put in the error response body? | 68k | 9.2k | 100% | 100% |
| Notion: How many requests per minute does the Notion API allow per connection on plans other than Business and Enterprise? | 29k | 6.0k | 100% | 100% |
| Notion: I'm writing long paragraphs into a Notion page through the API. What's the maximum number of characters a single rich text object's text content can have in a request? | 49k | 16k | 100% | 100% |
| Notion: What's the biggest file I can send to the Notion API in a single-part upload before I have to switch to a multi-part upload? | 71k | 32k | 100% | 100% |
| Question | Without WebMCP: time | With WebMCP: time | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Notion: I'm paginating through results from the Notion API. What's the largest page_size I can request per call on the list endpoints? | 15.8s | 7.6s | 100% | 100% |
| Notion: When my Notion integration sends too many requests, what error code string does the API put in the error response body? | 23.0s | 11.5s | 100% | 100% |
| Notion: How many requests per minute does the Notion API allow per connection on plans other than Business and Enterprise? | 13.5s | 7.0s | 100% | 100% |
| Notion: I'm writing long paragraphs into a Notion page through the API. What's the maximum number of characters a single rich text object's text content can have in a request? | 20.5s | 13.3s | 100% | 100% |
| Notion: What's the biggest file I can send to the Notion API in a single-part upload before I have to switch to a multi-part upload? | 23.1s | 18.6s | 100% | 100% |
| Question | Without WebMCP: steps | With WebMCP: steps | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Notion: I'm paginating through results from the Notion API. What's the largest page_size I can request per call on the list endpoints? | 6.7 | 3.0 | 100% | 100% |
| Notion: When my Notion integration sends too many requests, what error code string does the API put in the error response body? | 8.0 | 4.7 | 100% | 100% |
| Notion: How many requests per minute does the Notion API allow per connection on plans other than Business and Enterprise? | 5.3 | 3.0 | 100% | 100% |
| Notion: I'm writing long paragraphs into a Notion page through the API. What's the maximum number of characters a single rich text object's text content can have in a request? | 7.3 | 4.3 | 100% | 100% |
| Notion: What's the biggest file I can send to the Notion API in a single-part upload before I have to switch to a multi-part upload? | 8.7 | 7.0 | 100% | 100% |
| Question | Without WebMCP: cost | With WebMCP: cost | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Notion: I'm paginating through results from the Notion API. What's the largest page_size I can request per call on the list endpoints? | $0.016 | $0.0033 | 100% | 100% |
| Notion: When my Notion integration sends too many requests, what error code string does the API put in the error response body? | $0.026 | $0.0079 | 100% | 100% |
| Notion: How many requests per minute does the Notion API allow per connection on plans other than Business and Enterprise? | $0.016 | $0.0048 | 100% | 100% |
| Notion: I'm writing long paragraphs into a Notion page through the API. What's the maximum number of characters a single rich text object's text content can have in a request? | $0.022 | $0.0097 | 100% | 100% |
| Notion: What's the biggest file I can send to the Notion API in a single-part upload before I have to switch to a multi-part upload? | $0.027 | $0.017 | 100% | 100% |
The 5 WebMCP tools Notion registers
as the agent saw them · 2026-11-08
| Tool | Description | Called |
|---|---|---|
search_docs | Search the Notion Docs documentation for relevant pages and content. search_docs({"query":"pagination page_size"}) search_docs({"query":"rate limit error code"}) search_docs({"query":"errors status code rate limited"}) search_docs({"query":"rate limits requests per minute"}) search_docs({"query":"rich text maximum characters limit"}) search_docs({"query":"\"Size limits\" \"text.content\""}) search_docs({"query":"upload single-part multi-part file"}) search_docs({"query":"working-with-files-and-media single_part"}) | Called |
read_page | Read the full content of a Notion Docs documentation page as markdown. read_page({"path":"https://developers.notion.com/reference/pagination"}) read_page({"path":"https://developers.notion.com/reference/status-codes"}) read_page({"path":"https://developers.notion.com/reference/request-limits"}) read_page({"path":"https://developers.notion.com/reference/request-limits"}) read_page({"path":"https://developers.notion.com/reference/file-upload"}) read_page({"path":"https://developers.notion.com/guides/data-apis/sending-larger-files"}) read_page({"path":"https://developers.notion.com/guides/data-apis/uploading-small-files"}) read_page({"path":"https://developers.notion.com/guides/data-apis/working-with-files-and-media"}) | Called |
get_site_overview | Get an outline of the Notion Docs documentation, grouped by section, with a short description of each page. May be truncated for large sites; use search_docs to find pages not listed. | –Not called |
navigate_page | Navigate the current browser tab to a Notion Docs documentation page. | –Not called |
read_skill | Read a Notion Docs agent skill document by name. | –Not called |
- gemini-3.8-flash, reasoning medium, headless Chrome. Same prompt and step limit in both arms; 3 runs per question and arm.
- Without WebMCP: a DOM-snapshot agent that reads, clicks and fills.
- With WebMCP: only the tools the page registers (its own, or webmcpx’s on our test sites).
- Graded by facts in the answer and page state. No LLM judge.
- A live site: it ships its own WebMCP tools.
- 30 questions on 6 sites, 27 of them read-only; the biggest savings come from the sites' own search tools.
- 3 runs per question and arm: 3 of 3 solved is evidence, not proof of 100%.
- Time is the agent loop after the page loaded; cost is the model's API usage at dated list prices, cache included.
- Every failed run says why in the results (bot check, answer not from the page, step limit…).
git clone https://github.com/webmcpx/webmcpx
cd webmcpx && just setup
GEMINI_API_KEY=... just bench --model gemini-3.8-flash --reasoning-effort mediumKiln & Co local
“Put the most expensive teapot you sell in my cart.”
5.9×
fewer tokens
12k → 2.1k tokens
| Question | Without WebMCP: tokens | With WebMCP: tokens | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Kiln & Co (local test site): Put the most expensive teapot you sell in my cart. | 12k | 2.1k | 100% | 100% |
| Kiln & Co (local test site): Add one Stoneware Teapot and one Speckled Bowl to my cart, then tell me the cart total. | 16k | 2.9k | 100% | 100% |
| Kiln & Co (local test site): Add two of the cheapest blue mug to my cart. | 10k | 2.2k | 100% | 100% |
| Kiln & Co (local test site): I need a green gift under $30 that I can buy right now. What do you have? | 7.3k | 2.6k | 100% | 100% |
| Question | Without WebMCP: time | With WebMCP: time | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Kiln & Co (local test site): Put the most expensive teapot you sell in my cart. | 15.3s | 7.2s | 100% | 100% |
| Kiln & Co (local test site): Add one Stoneware Teapot and one Speckled Bowl to my cart, then tell me the cart total. | 14.0s | 8.0s | 100% | 100% |
| Kiln & Co (local test site): Add two of the cheapest blue mug to my cart. | 12.1s | 6.1s | 100% | 100% |
| Kiln & Co (local test site): I need a green gift under $30 that I can buy right now. What do you have? | 11.7s | 9.0s | 100% | 100% |
| Question | Without WebMCP: steps | With WebMCP: steps | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Kiln & Co (local test site): Put the most expensive teapot you sell in my cart. | 6.0 | 3.0 | 100% | 100% |
| Kiln & Co (local test site): Add one Stoneware Teapot and one Speckled Bowl to my cart, then tell me the cart total. | 6.0 | 3.3 | 100% | 100% |
| Kiln & Co (local test site): Add two of the cheapest blue mug to my cart. | 5.0 | 3.0 | 100% | 100% |
| Kiln & Co (local test site): I need a green gift under $30 that I can buy right now. What do you have? | 4.0 | 3.0 | 100% | 100% |
| Question | Without WebMCP: cost | With WebMCP: cost | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Kiln & Co (local test site): Put the most expensive teapot you sell in my cart. | $0.01 | $0.0022 | 100% | 100% |
| Kiln & Co (local test site): Add one Stoneware Teapot and one Speckled Bowl to my cart, then tell me the cart total. | $0.013 | $0.0031 | 100% | 100% |
| Kiln & Co (local test site): Add two of the cheapest blue mug to my cart. | $0.0092 | $0.0023 | 100% | 100% |
| Kiln & Co (local test site): I need a green gift under $30 that I can buy right now. What do you have? | $0.0069 | $0.0033 | 100% | 100% |
The 4 WebMCP tools Kiln & Co registers
as the agent saw them · 2026-11-08
| Tool | Description | Called |
|---|---|---|
search_products | Search the catalog. Every filter is optional; with none, returns every product. Prices are USD. search_products({"category":"teapot","sort":"price_desc"}) search_products({"query":"Stoneware Teapot"}) search_products({"query":"Speckled Bowl"}) search_products({"category":"mug","color":"blue","sort":"price_asc"}) search_products({"color":"green","max_price":30}) search_products({"color":"green"}) | Called |
add_to_cart | Add a product to the cart by sku. Adding a sku already in the cart increases its quantity. add_to_cart({"quantity":1,"sku":"TPT-02"}) add_to_cart({"quantity":1,"sku":"TPT-01"}) add_to_cart({"quantity":1,"sku":"BWL-01"}) add_to_cart({"quantity":2,"sku":"MUG-03"}) | Called |
get_cart | Return the cart: items with quantity and the total in USD. get_cart({}) | Called |
get_page_context | Describe the page the user is on: title, URL, language, meta description, heading outline and the text the user has selected. Use it for the page's structure or the user's selection; to find a fact on the page, call search_page instead. | –Not called |
- gemini-3.8-flash, reasoning medium, headless Chrome. Same prompt and step limit in both arms; 3 runs per question and arm.
- Without WebMCP: a DOM-snapshot agent that reads, clicks and fills.
- With WebMCP: only the tools the page registers (its own, or webmcpx’s on our test sites).
- Graded by facts in the answer and page state. No LLM judge.
- A local test site: its tools come from webmcpx.
- 30 questions on 6 sites, 27 of them read-only; the biggest savings come from the sites' own search tools.
- 3 runs per question and arm: 3 of 3 solved is evidence, not proof of 100%.
- Time is the agent loop after the page loaded; cost is the model's API usage at dated list prices, cache included.
- Every failed run says why in the results (bot check, answer not from the page, step limit…).
git clone https://github.com/webmcpx/webmcpx
cd webmcpx && just setup
GEMINI_API_KEY=... just bench --model gemini-3.8-flash --reasoning-effort mediumOrbit docs local
“Which environment variable makes Orbit read its config file from a different path?”
2.7×
fewer tokens
3.4k → 1.3k tokens
| Question | Without WebMCP: tokens | With WebMCP: tokens | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Orbit docs (local test site): Which environment variable makes Orbit read its config file from a different path? | 3.4k | 1.3k | 100% | 100% |
| Orbit docs (local test site): My CI step failed with orbit exit code 4. What does that mean? | 3.4k | 1.3k | 100% | 100% |
| Question | Without WebMCP: time | With WebMCP: time | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Orbit docs (local test site): Which environment variable makes Orbit read its config file from a different path? | 6.9s | 5.2s | 100% | 100% |
| Orbit docs (local test site): My CI step failed with orbit exit code 4. What does that mean? | 5.1s | 5.8s | 100% | 100% |
| Question | Without WebMCP: steps | With WebMCP: steps | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Orbit docs (local test site): Which environment variable makes Orbit read its config file from a different path? | 2.0 | 2.0 | 100% | 100% |
| Orbit docs (local test site): My CI step failed with orbit exit code 4. What does that mean? | 2.0 | 2.0 | 100% | 100% |
| Question | Without WebMCP: cost | With WebMCP: cost | Without WebMCP: solved | With WebMCP: solved |
|---|---|---|---|---|
| Orbit docs (local test site): Which environment variable makes Orbit read its config file from a different path? | $0.003 | $0.0013 | 100% | 100% |
| Orbit docs (local test site): My CI step failed with orbit exit code 4. What does that mean? | $0.0031 | $0.0015 | 100% | 100% |
The 3 WebMCP tools Orbit docs registers
as the agent saw them · 2026-11-08
| Tool | Description | Called |
|---|---|---|
search_page | Find passages on the current page that match a query. Returns up to `limit` passages ranked by relevance, each with its section heading and a link to it. Prefer it to reading the whole page when looking for something specific. search_page({"query":"environment variable config path"}) search_page({"query":"exit code 4"}) | Called |
fetch_as_markdown | Return the main content of the current page as Markdown. Long pages come in parts: when `truncated` is true, call again with `cursor` set to `next_cursor`. | –Not called |
get_page_context | Describe the page the user is on: title, URL, language, meta description, heading outline and the text the user has selected. Use it for the page's structure or the user's selection; to find a fact on the page, call search_page instead. | –Not called |
- gemini-3.8-flash, reasoning medium, headless Chrome. Same prompt and step limit in both arms; 3 runs per question and arm.
- Without WebMCP: a DOM-snapshot agent that reads, clicks and fills.
- With WebMCP: only the tools the page registers (its own, or webmcpx’s on our test sites).
- Graded by facts in the answer and page state. No LLM judge.
- A local test site: its tools come from webmcpx.
- 30 questions on 6 sites, 27 of them read-only; the biggest savings come from the sites' own search tools.
- 3 runs per question and arm: 3 of 3 solved is evidence, not proof of 100%.
- Time is the agent loop after the page loaded; cost is the model's API usage at dated list prices, cache included.
- Every failed run says why in the results (bot check, answer not from the page, step limit…).
git clone https://github.com/webmcpx/webmcpx
cd webmcpx && just setup
GEMINI_API_KEY=... just bench --model gemini-3.8-flash --reasoning-effort medium