repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

extensions/web/README.md

Raw

web

Web fetch, search, site-map, and image-analysis tools for Pi.

Install / load

Loaded through the root pi-ext package. See ../../README.md.

Commands / tools / settings

Commands: none.

Tools:

Hooks:

Settings: none registered.

External binaries used when needed:

Provider/env config read by source:

Bench

Run mise run bench here, or mise //extensions/web:bench from the repository root, to benchmark web processing hot paths.

Behavior

Markdown source
# web

Web fetch, search, site-map, and image-analysis tools for Pi.

## Install / load

Loaded through the root `pi-ext` package.
See [../../README.md](../../README.md).

## Commands / tools / settings

Commands:
none.

Tools:

- `web_fetch`
  - `url`:
    URL to fetch.
  - `format`:
    optional; `markdown` default, or `text`, `html`, `json`, `raw`.
  - `maxChars`:
    optional; default `100000`, min `100`, max `1000000`.
  - `timeout`:
    optional; declared by schema.
    Current `web_fetch` execution uses the Pi/tool abort signal and does not
    pass this value to `fetchUrlStream`.
  - `linesMatching`:
    optional; returns matching line excerpts instead of the full visible output.
  - `contextLines`:
    optional; line context around `linesMatching` hits.
- `web_map`
  - discovers site maps and related links.
- `web_search`
  - `query`:
    search query.
  - `count`:
    optional; default `5`, max `20`.
  - `format`:
    optional; `text` default, or `json`.
  - `timeout`:
    optional; default `120000` ms, min `5000`, max `600000`.
- `understand_image`
  - `prompt`:
    image question or extraction request.
  - `image_source`:
    HTTP/HTTPS URL, local path, `@`-prefixed local path, or data URL.

Hooks:

- `session_shutdown`:
  removes temp dirs created for fetched binary/image/media/PDF/office content.

Settings:
none registered.

External binaries used when needed:

- `pandoc`:
  HTML conversion; DOCX/ODT/EPUB/RTF extraction.
  Missing binary emits a warning notification on session start.
- `pdftotext`:
  PDF extraction.
  Missing binary emits a warning notification on session start.
- `identify`:
  image metadata.
  Missing binary emits an info notification on session start.
  Missing/failing binary returns fallback metadata text.
- `ffprobe`:
  audio/video metadata.
  Missing binary emits an info notification on session start.
  Missing/failing binary returns fallback metadata text.

Provider/env config read by source:

- Z.AI search key:
  Pi model registry providers `zai`, `z-ai`, `z.ai`, `bigmodel`; fallback env
  `ZAI_API_KEY`, `Z_AI_API_KEY`.
- Z.AI search endpoint/options:
  `ZAI_BASE_URL`, `ZAI_SEARCH_MCP_URL`, `ZAI_SEARCH_BASE_URL`,
  `ZAI_SEARCH_FRESHNESS`, `ZAI_SEARCH_CONTENT_SIZE`, `ZAI_SEARCH_LOCATION`.
- MiniMax key:
  Pi model registry provider `minimax`; fallback env `MINIMAX_API_KEY`.
- MiniMax base URL:
  model registry providers `minimax`, `minimax-cn`; fallback env
  `MINIMAX_API_HOST`.
- OpenAI Codex search auth:
  Pi model registry provider `openai-codex` OAuth.

## Bench

Run `mise run bench` here, or `mise //extensions/web:bench` from the repository root, to benchmark web processing hot paths.

## Behavior

- `web_fetch` uses native `fetch` with browser-like headers and a per-host
  referer.
- `web_fetch` accepts only HTTP and HTTPS URLs.
  Private/reserved IP targets, blocked DNS answers, and unsafe redirects are
  rejected.
- Fetch output is stored in the Pi state directory with a `responseId` and
  `fullOutputPath`.
  Tool-visible text is capped; full output stays on disk.
- `linesMatching` returns numbered excerpts with optional surrounding context.
- Content routing is based on `Content-Type`:
  - HTML/XHTML:
    strips common prompt-injection delivery vectors, extracts main content,
    converts with `pandoc` and `llm-extract.lua`.
  - Text/XML/YAML/CSV/TSV:
    returns UTF-8 text.
  - JSON:
    pretty-prints when parseable, otherwise returns raw text.
  - PDF:
    writes temp file, runs `pdftotext -layout`.
  - DOCX/ODT/EPUB/RTF:
    writes temp file, runs `pandoc` to plain text.
  - Images:
    writes temp file, returns metadata from `identify`, plus inline image
    content.
  - Audio/video:
    writes temp file, returns summary and `ffprobe` JSON.
  - Other binary:
    writes temp file and returns path plus content type.
- `web_fetch` `raw` format skips conversion, decodes the body as UTF-8, and
  truncates to `maxChars`.
- HTML cleanup removes comments, scripts, styles, iframes, objects, templates,
  noscript blocks, embeds, hidden inputs, long meta content,
  hidden/visually-hidden elements, and non-image `aria-label`/`alt`/`title`
  attrs.
- Main-content extraction prefers `<article>`, `#mw-content-text`,
  `#docContent`, `#content`, `<main>`, then `role="main"`; fallback is full
  page.
- Converted fetch text, raw fetch text, and search output are wrapped as
  untrusted external data.
  Image, media, and binary early-return paths are not wrapped by the current
  implementation.
- `web_map` discovers site maps from robots.txt, sitemap XML, and llms links.
- `web_search` provider order:
  OpenAI Codex native search when authenticated, then Z.AI Coding Plan MCP when
  configured, then MiniMax search when configured, then DuckDuckGo HTML.
- Z.AI calls MCP tool `web_search_prime`; `ZAI_SEARCH_FRESHNESS` accepts `day`,
  `week`, `month`, `year`, or no-limit values.
- DuckDuckGo fallback throttles requests to at least 2 seconds apart and reports
  captcha/dedup diagnostics.
- MiniMax search posts to `/v1/coding_plan/search`.
- `understand_image` sends the prompt and image data to MiniMax VLM.
  It accepts HTTP/HTTPS URLs, data URLs, and local files.
  Local file data is base64 encoded before the request.