3 -Web fetch, search, site-map, and image-analysis tools for Pi.
3 +One web tool for Pi: fetch, search, and site-map discovery.
15 +Tool: `web`, with required `action: fetch | map | search` and flat action-specific arguments.
16 +Invalid or irrelevant arguments are rejected before I/O.
17 +The former separate tools and MiniMax image analysis are removed, without aliases.
18 +Update tool allowlists to `web`.
21 26 optional; `markdown` default, or `text`, `html`, `json`, `raw`.
23 - optional; default `100000`, min `100`, max `1000000`.
28 + optional; default/max `20000`, min `100`.
25 - optional; declared by schema.
26 - Current `web_fetch` execution uses the Pi/tool abort signal and does not
27 - pass this value to `fetchUrlStream`.
30 + optional; default `120000` ms, min `5000`, max `600000`.
31 + Passed to `fetchUrlStream` alongside the Pi/tool abort signal.
29 33 optional; returns matching line excerpts instead of the full visible output.
31 35 optional; line context around `linesMatching` hits.
33 - - discovers site maps and related links.
38 + site URL; discovers site maps and related links without fetching page bodies.
40 + optional; default `1000`, min `1`, max `10000`.
42 + optional; default `20`, min `1`, max `100`.
44 + optional; default `8000`, min `100`, max `100000`.
40 51 optional; `text` default, or `json`.
42 53 optional; default `120000` ms, min `5000`, max `600000`.
45 - image question or extraction request.
47 - HTTP/HTTPS URL, local path, `@`-prefixed local path, or data URL.
92 98 Run `mise run bench` here, or `mise //extensions/web:bench` from the repository root, to benchmark web processing hot paths.
100 +##
Live search comparison 102 +Run `mise run //extensions/web:evaluate-search` to compare all configured providers on five fixed English/German technical and consumer-information queries.
103 +It uses existing Pi credentials through ModelRegistry, makes no model-catalog refresh, applies a 45-second deadline per provider, rotates provider order, and records target rank, elapsed time, and full results in the Pi state directory.
104 +Providers with authentication, quota, or CAPTCHA failures are not retried during that run.
105 +These small-sample target-rank checks are not a general search-quality benchmark.
96 -- `web_fetch` uses native `fetch` with browser-like headers and a per-host
109 +- `web` with `action: fetch` uses native `fetch` with browser-like headers and a per-host
98 -- `web_fetch` accepts only HTTP and HTTPS URLs.
111 +- Fetch accepts only HTTP and HTTPS URLs.
99 112 Private/reserved IP targets, blocked DNS answers, and unsafe redirects are
101 114 - Fetch output is stored in the Pi state directory with a `responseId` and
102 115 `fullOutputPath`.
103 - Tool-visible text is capped; full output stays on disk.
116 + Capped output, matching excerpts, and no-match results include the path in
117 + tool-visible text so the complete saved output can be read directly.
104 118 - `linesMatching` returns numbered excerpts with optional surrounding context.
105 119 - Content routing is based on `Content-Type`:
121 135 writes temp file, returns summary and `ffprobe` JSON.
123 137 writes temp file and returns path plus content type.
124 -- `web_fetch` `raw` format skips conversion, decodes the body as UTF-8, and
138 +- Fetch `raw` format skips conversion, decodes the body as UTF-8, and
125 139 truncates to `maxChars`.
126 140 - HTML cleanup removes comments, scripts, styles, iframes, objects, templates,
127 141 noscript blocks, embeds, hidden inputs, long meta content,
134 148 untrusted external data.
135 149 Image, media, and binary early-return paths are not wrapped by the current
137 -- `web_map` discovers site maps from robots.txt, sitemap XML, and llms links.
138 -- `web_search` provider order:
139 - OpenAI Codex native search when authenticated, then Z.AI Coding Plan MCP when
140 - configured, then MiniMax search when configured, then DuckDuckGo HTML.
151 +- `web` with `action: map` discovers site maps from robots.txt, sitemap XML, and llms links.
152 + Its URL listing shows at most 50 entries and obeys `maxChars`.
153 + If either limit omits URLs, the result identifies the listing as partial and
154 + includes the saved map's `fullOutputPath`.
155 +- `web` with `action: search` provider order:
156 + Google WML, anonymous Parallel, OpenAI Codex native search when authenticated, MiniMax search when configured, DuckDuckGo HTML, then Z.AI Coding Plan MCP when configured.
157 + Cancellation stops fallback.
158 +- Parallel uses the anonymous hosted MCP at `https://search.parallel.ai/mcp` without API keys, model metadata, or conversation identifiers.
159 + All excerpts from each selected result are preserved as its snippet, without character caps.
160 + The existing result-count limit still applies.
161 +- Google uses `/wml/search` with a fixed Nokia user agent and native HTTP, without JavaScript or browser automation.
162 + Requests reserve slots at least two seconds apart.
163 + CAPTCHA, consent pages, and unrecognized layouts fail into the next provider; no CAPTCHA solving or proxy rotation.
164 + This is an undocumented endpoint and can stop working.
165 +- Parallel and Z.AI share the MCP transport in `mcp.ts`: JSON/SSE decoding, response-ID matching, initialization, session headers, protocol negotiation, and redirect rejection.
166 + Authentication, tool arguments, result decoding, and provider-specific errors remain separate.
141 167 - Z.AI calls MCP tool `web_search_prime`; `ZAI_SEARCH_FRESHNESS` accepts `day`,
142 168 `week`, `month`, `year`, or no-limit values.
143 -- DuckDuckGo fallback throttles requests to at least 2 seconds apart and reports
144 - captcha/dedup diagnostics.
169 +- DuckDuckGo submits the HTML no-JavaScript form using POST, query/region fields, and origin/referer headers, instead of searching with GET.
170 + It throttles requests to at least two seconds apart, includes the throttle wait in its deadline, and reports CAPTCHA as a provider failure.
171 + A local two-query comparison returned ten results per query for HTML/Lite POST, while both GET interfaces returned CAPTCHA with no results.
172 + A follow-up five-query evaluation through the production HTML provider found the intended target first for every query.
173 + This fixes the observed request pattern, not all possible DDG blocks; no CAPTCHA solver, proxy rotation, or browser dependency is added.
174 + See [DDG's non-JavaScript interfaces](https://duckduckgo.com/duckduckgo-help-pages/features/non-javascript) and [SearXNG's request and bot-blocker notes](https://docs.searxng.org/dev/engines/online/duckduckgo.html).
145 175 - MiniMax search posts to `/v1/coding_plan/search`.
146 -- `understand_image` sends the prompt and image data to MiniMax VLM.
147 - It accepts HTTP/HTTPS URLs, data URLs, and local files.
148 - Local file data is base64 encoded before the request.