MiniMax key:
Pi model registry provider minimax; fallback env MINIMAX_API_KEY.
MiniMax base URL:
model registry providers minimax, minimax-cn; fallback env
MINIMAX_API_HOST.
OpenAI Codex search auth:
Pi model registry provider openai-codex OAuth.
Bench
Run mise run bench here, or mise //extensions/web:bench from the repository root, to benchmark web processing hot paths.
Behavior
web_fetch uses native fetch with browser-like headers and a per-host
referer.
web_fetch accepts only HTTP and HTTPS URLs.
Private/reserved IP targets, blocked DNS answers, and unsafe redirects are
rejected.
Fetch output is stored in the Pi state directory with a responseId and
fullOutputPath.
Tool-visible text is capped; full output stays on disk.
linesMatching returns numbered excerpts with optional surrounding context.
Content routing is based on Content-Type:
HTML/XHTML:
strips common prompt-injection delivery vectors, extracts main content,
converts with pandoc and llm-extract.lua.
Text/XML/YAML/CSV/TSV:
returns UTF-8 text.
JSON:
pretty-prints when parseable, otherwise returns raw text.
PDF:
writes temp file, runs pdftotext -layout.
DOCX/ODT/EPUB/RTF:
writes temp file, runs pandoc to plain text.
Images:
writes temp file, returns metadata from identify, plus inline image
content.
Audio/video:
writes temp file, returns summary and ffprobe JSON.
Other binary:
writes temp file and returns path plus content type.
web_fetchraw format skips conversion, decodes the body as UTF-8, and
truncates to maxChars.
HTML cleanup removes comments, scripts, styles, iframes, objects, templates,
noscript blocks, embeds, hidden inputs, long meta content,
hidden/visually-hidden elements, and non-image aria-label/alt/title
attrs.
Main-content extraction prefers <article>, #mw-content-text,
#docContent, #content, <main>, then role="main"; fallback is full
page.
Converted fetch text, raw fetch text, and search output are wrapped as
untrusted external data.
Image, media, and binary early-return paths are not wrapped by the current
implementation.
web_map discovers site maps from robots.txt, sitemap XML, and llms links.
web_search provider order:
OpenAI Codex native search when authenticated, then Z.AI Coding Plan MCP when
configured, then MiniMax search when configured, then DuckDuckGo HTML.
DuckDuckGo fallback throttles requests to at least 2 seconds apart and reports
captcha/dedup diagnostics.
MiniMax search posts to /v1/coding_plan/search.
understand_image sends the prompt and image data to MiniMax VLM.
It accepts HTTP/HTTPS URLs, data URLs, and local files.
Local file data is base64 encoded before the request.
Markdown source
# webWeb fetch, search, site-map, and image-analysis tools for Pi.
## Install / loadLoaded through the root `pi-ext` package.
See [../../README.md](../../README.md).
## Commands / tools / settingsCommands:
none.
Tools:
- `web_fetch`
- `url`:
URL to fetch.
- `format`:
optional; `markdown` default, or `text`, `html`, `json`, `raw`.
- `maxChars`:
optional; default `100000`, min `100`, max `1000000`.
- `timeout`:
optional; declared by schema.
Current `web_fetch` execution uses the Pi/tool abort signal and does not
pass this value to `fetchUrlStream`.
- `linesMatching`:
optional; returns matching line excerpts instead of the full visible output.
- `contextLines`:
optional; line context around `linesMatching` hits.
- `web_map`
- discovers site maps and related links.
- `web_search`
- `query`:
search query.
- `count`:
optional; default `5`, max `20`.
- `format`:
optional; `text` default, or `json`.
- `timeout`:
optional; default `120000` ms, min `5000`, max `600000`.
- `understand_image`
- `prompt`:
image question or extraction request.
- `image_source`:
HTTP/HTTPS URL, local path, `@`-prefixed local path, or data URL.
Hooks:
- `session_shutdown`:
removes temp dirs created for fetched binary/image/media/PDF/office content.
Settings:
none registered.
External binaries used when needed:
- `pandoc`:
HTML conversion; DOCX/ODT/EPUB/RTF extraction.
Missing binary emits a warning notification on session start.
- `pdftotext`:
PDF extraction.
Missing binary emits a warning notification on session start.
- `identify`:
image metadata.
Missing binary emits an info notification on session start.
Missing/failing binary returns fallback metadata text.
- `ffprobe`:
audio/video metadata.
Missing binary emits an info notification on session start.
Missing/failing binary returns fallback metadata text.
Provider/env config read by source:
- Z.AI search key:
Pi model registry providers `zai`, `z-ai`, `z.ai`, `bigmodel`; fallback env
`ZAI_API_KEY`, `Z_AI_API_KEY`.
- Z.AI search endpoint/options:
`ZAI_BASE_URL`, `ZAI_SEARCH_MCP_URL`, `ZAI_SEARCH_BASE_URL`,
`ZAI_SEARCH_FRESHNESS`, `ZAI_SEARCH_CONTENT_SIZE`, `ZAI_SEARCH_LOCATION`.
- MiniMax key:
Pi model registry provider `minimax`; fallback env `MINIMAX_API_KEY`.
- MiniMax base URL:
model registry providers `minimax`, `minimax-cn`; fallback env
`MINIMAX_API_HOST`.
- OpenAI Codex search auth:
Pi model registry provider `openai-codex` OAuth.
## BenchRun `mise run bench` here, or `mise //extensions/web:bench` from the repository root, to benchmark web processing hot paths.
## Behavior- `web_fetch` uses native `fetch` with browser-like headers and a per-host
referer.
- `web_fetch` accepts only HTTP and HTTPS URLs.
Private/reserved IP targets, blocked DNS answers, and unsafe redirects are
rejected.
- Fetch output is stored in the Pi state directory with a `responseId` and
`fullOutputPath`.
Tool-visible text is capped; full output stays on disk.
- `linesMatching` returns numbered excerpts with optional surrounding context.
- Content routing is based on `Content-Type`:
- HTML/XHTML:
strips common prompt-injection delivery vectors, extracts main content,
converts with `pandoc` and `llm-extract.lua`.
- Text/XML/YAML/CSV/TSV:
returns UTF-8 text.
- JSON:
pretty-prints when parseable, otherwise returns raw text.
- PDF:
writes temp file, runs `pdftotext -layout`.
- DOCX/ODT/EPUB/RTF:
writes temp file, runs `pandoc` to plain text.
- Images:
writes temp file, returns metadata from `identify`, plus inline image
content.
- Audio/video:
writes temp file, returns summary and `ffprobe` JSON.
- Other binary:
writes temp file and returns path plus content type.
- `web_fetch` `raw` format skips conversion, decodes the body as UTF-8, and
truncates to `maxChars`.
- HTML cleanup removes comments, scripts, styles, iframes, objects, templates,
noscript blocks, embeds, hidden inputs, long meta content,
hidden/visually-hidden elements, and non-image `aria-label`/`alt`/`title`
attrs.
- Main-content extraction prefers `<article>`, `#mw-content-text`,
`#docContent`, `#content`, `<main>`, then `role="main"`; fallback is full
page.
- Converted fetch text, raw fetch text, and search output are wrapped as
untrusted external data.
Image, media, and binary early-return paths are not wrapped by the current
implementation.
- `web_map` discovers site maps from robots.txt, sitemap XML, and llms links.
- `web_search` provider order:
OpenAI Codex native search when authenticated, then Z.AI Coding Plan MCP when
configured, then MiniMax search when configured, then DuckDuckGo HTML.
- Z.AI calls MCP tool `web_search_prime`; `ZAI_SEARCH_FRESHNESS` accepts `day`,
`week`, `month`, `year`, or no-limit values.
- DuckDuckGo fallback throttles requests to at least 2 seconds apart and reports
captcha/dedup diagnostics.
- MiniMax search posts to `/v1/coding_plan/search`.
- `understand_image` sends the prompt and image data to MiniMax VLM.
It accepts HTTP/HTTPS URLs, data URLs, and local files.
Local file data is base64 encoded before the request.