repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

Merge research workflow into wiki skill

2c13597d3c96adc320178a6f49c95f4f66ca33d0

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (3)

prompts/wiki.md

Added
new file mode 100644
1 +---
2 +description: "Research + persistent knowledge base. init|recall|ingest|retro|lint|status|digest. No args = recall with focus."
3 +argument-hint: "[init|recall|ingest|retro|lint|status|digest] [focus|source|query]"
4 +section: Knowledge
5 +topLevelCli: true
6 +---
8 +# /wiki
10 +Use the `wiki` skill; do not assume a filesystem install path.
12 +Parse `$ARGUMENTS` for a subcommand and focus:
14 +- No args / plain text → **recall**: search wiki for relevant pages and raw sources, synthesize answer. The text is the query.
15 +- `init [topic]` → **init**: bootstrap `~/Notes/Wiki/` if missing, set topic/schema.
16 +- `recall [query]` → **recall**: search compiled pages + raw sources + legacy `~/.pi/research/`, then synthesize.
17 +- `ingest <source>` → **ingest**: register + capture + compile source. Source can be URL, repo URL, archive URL, file path, directory, pasted text, docs/code/concepts, or old `~/.pi/research` item. Use `web_fetch` for readable URLs when available; use `git_clone_safe` for repos when available.
18 +- `retro [insight]` → **retro**: save an atomic insight from the current session as a wiki page.
19 +- `lint` → **lint**: health check the wiki.
20 +- `status` → **status**: quick stats.
21 +- `digest [daily|weekly]` → **digest**: summarize recent changes.
23 +Follow the skill's rules for each operation. New knowledge lives under `~/Notes/Wiki/`; old `~/.pi/research/` is legacy cache only.

skills/research/SKILL.md

Modified
2 2 name: research
3 -description: Research via web downloads and git clones. Fetch docs, code, concepts. Use curl or monolith for pages, git sparse/shallow for repos. Persisted to state folder.
3 +description: "Compatibility alias for wiki research. Trigger on research, web downloads, git/repo research, fetching docs/code/concepts, curl/monolith pages, sparse/shallow repos, persisted research state. Use wiki instead."
6 -# Research Skill
8 -Web and repository research.
9 -Download, clone, extract, persist.
11 -## State Folder
13 -Research data lives in `~/.local/state/pi/research/` — **never delete after use**.
15 -```text
16 -~/.local/state/pi/research/
17 -├── pages/ # web page downloads (HTML)
18 -├── repos/ # git clones
19 -└── extracts/ # extracted archives
20 -```
22 -### Create Structure (Important: Use Correct Syntax)
24 -```bash
25 -# CORRECT: separate mkdir commands
26 -mkdir -p ~/.local/state/pi/research/pages
27 -mkdir -p ~/.local/state/pi/research/repos
28 -mkdir -p ~/.local/state/pi/research/extracts
30 -# WRONG: brace expansion may create literal dir named "{pages,repos,extracts}"
31 -# mkdir -p ~/.local/state/pi/research/{pages,repos,extracts} # AVOID THIS
32 -```
34 -## Core Workflow
36 -```text
37 -1. Check ~/.local/state/pi/research/ — maybe already downloaded
38 -2. If not → download/clone to appropriate subfolder
39 -3. Search within local files first
40 -4. Return findings
41 -```
43 -## Downloading Web Pages
45 -### Primary: monolith (Recommended)
47 -monolith saves pages as standalone HTML — no external deps, works offline.
49 -```bash
50 -# Basic
51 -monolith https://example.com -o ~/.local/state/pi/research/pages/example.com.html
53 -# Clean extract (remove JS, CSS, fonts, images, audio, video, metadata)
54 -monolith -a -c -F -i -j -M -I https://example.com -o ~/.local/state/pi/research/pages/example.com.clean.html
56 -# Quiet mode (less output)
57 -monolith -q https://example.com -o ~/.local/state/pi/research/pages/example.com.html
59 -# Custom timeout (for slow sites)
60 -monolith -t 120 https://example.com -o ~/.local/state/pi/research/pages/example.com.html
61 -```
63 -### Fallback: curl
65 -Use when monolith fails or for simple GET requests.
67 -```bash
68 -# Basic download
69 -curl -sL "https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
71 -# With better naming (domain + path)
72 -curl -sL "https://docs.framework.com/guide" -o ~/.local/state/pi/research/pages/docs.framework.com.guide.html
73 -```
75 -**curl flags:**
77 -- `-s` = silent (no progress meter)
78 -- `-L` = follow redirects
79 -- `-o file` = output to file
81 -**monolith flags for clean extract:**
83 -- `-a` = no audio
84 -- `-c` = no CSS
85 -- `-F` = no fonts
86 -- `-i` = no images
87 -- `-j` = no JavaScript
88 -- `-M` = no metadata
89 -- `-I` = isolate (no external refs)
91 -### Which Method to Use?
93 -| Situation | Use |
94 -| ----------------------------------- | ------------------------------- |
95 -| Most pages | `monolith` (standalone HTML) |
96 -| Complex page (SPA, dynamic content) | `monolith` |
97 -| Readability focused | `monolith -a -c -F -i -j -M -I` |
98 -| Quick text fetch | `curl` |
99 -| API response (JSON) | `curl` |
101 -### Naming Convention
103 -```text
104 -{domain}.{path}.{suffix}.html
105 -example.com.docs.html
106 -github.com.user.repo.readme.html
107 -docs.framework.com.api.v2.html
108 -```
110 -## Git Repositories
112 -### Shallow Clone (single branch, history truncated)
114 -```bash
115 -git clone --depth 1 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
116 -```
118 -### Shallow Clone Specific Branch
120 -```bash
121 -git clone --depth 1 --branch main https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
122 -```
124 -### Sparse Clone (only wanted directories)
126 -```bash
127 -# Step 1: Init
128 -git init ~/.local/state/pi/research/repos/github.com.user.repo
129 -cd ~/.local/state/pi/research/repos/github.com.user.repo
131 -# Step 2: Add remote
132 -git remote add origin https://github.com/user/repo
134 -# Step 3: Enable sparse checkout
135 -git sparse-checkout init --cone
137 -# Step 4: Specify paths to keep
138 -git sparse-checkout set src/docs
140 -# Step 5: Pull
141 -git pull origin main
142 -```
144 -### Sparse + Shallow Combined
146 -```bash
147 -git clone --depth 1 --filter=blob:none --sparse https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
148 -cd ~/.local/state/pi/research/repos/github.com.user.repo
149 -git sparse-checkout set src docs
150 -git checkout main
151 -```
153 -### Partial Clone (fetch only reachable objects)
155 -```bash
156 -git clone --depth 1 --filter=tree:0 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
157 -```
159 -### Repo Naming Convention
161 -```text
162 -{host}.{user}.{repo}[.{branch}][.{path}]
163 -github.com.user.library.main.html
164 -github.com.user.framework.src.html
165 -```
167 -## Search Patterns
169 -### Check Before Download
171 -```bash
172 -# List already downloaded pages
173 -ls ~/.local/state/pi/research/pages/
175 -# List already cloned repos
176 -ls ~/.local/state/pi/research/repos/
178 -# Find by name (grep)
179 -ls ~/.local/state/pi/research/pages/ | grep domain
180 -ls ~/.local/state/pi/research/repos/ | grep user
181 -```
183 -### Within Downloaded Pages
185 -```bash
186 -# Find pattern in single page
187 -grep -n "pattern" ~/.local/state/pi/research/pages/*.html
189 -# Find across all pages
190 -grep -rn "pattern" ~/.local/state/pi/research/pages/
192 -# Case-insensitive
193 -grep -ri "pattern" ~/.local/state/pi/research/pages/
195 -# With ripgrep (faster, if available)
196 -rg "pattern" ~/.local/state/pi/research/pages/
197 -```
199 -### Within Cloned Repos
201 -```bash
202 -# Find in specific files
203 -grep -rn "function_name" ~/.local/state/pi/research/repos/github.com.user.repo/src/
205 -# Find by file extension
206 -find ~/.local/state/pi/research/repos/github.com.user.repo -name "*.py" -exec grep -l "pattern" {} \;
208 -# Find and show context
209 -grep -rn -A2 -B2 "pattern" ~/.local/state/pi/research/repos/github.com.user.repo/
211 -# With ripgrep
212 -rg -A2 -B2 "pattern" ~/.local/state/pi/research/repos/github.com.user.repo/
213 -```
215 -### Search Git History (if full clone needed)
217 -```bash
218 -cd ~/.local/state/pi/research/repos/github.com.user.repo
220 -git log --oneline -20
221 -git log --all --oneline --graph
222 -git log -p --follow -- filename
223 -```
225 -### Across Everything
227 -```bash
228 -# All pages
229 -rg "pattern" ~/.local/state/pi/research/pages/
231 -# All repos
232 -rg "pattern" ~/.local/state/pi/research/repos/
234 -# Everything
235 -rg "pattern" ~/.local/state/pi/research/
236 -```
238 -## Extract Archives
240 -```bash
241 -# Download and extract tar.gz
242 -curl -sL https://example.com/archive.tar.gz | tar -xz -C ~/.local/state/pi/research/extracts/
244 -# Download and extract zip
245 -curl -sL https://example.com/archive.zip -o /tmp/archive.zip
246 -unzip /tmp/archive.zip -d ~/.local/state/pi/research/extracts/
248 -# Or download first
249 -wget -O ~/.local/state/pi/research/extracts/archive.tar.gz https://example.com/archive.tar.gz
250 -tar -xzf ~/.local/state/pi/research/extracts/archive.tar.gz -C ~/.local/state/pi/research/extracts/
251 -```
253 -## Error Handling
255 -### monolith Failures
257 -```bash
258 -# Longer timeout
259 -monolith -t 120 https://slow.site.com -o ~/.local/state/pi/research/pages/site.com.html
261 -# Ignore network errors
262 -monolith -e https://maybe-down.site.com -o ~/.local/state/pi/research/pages/site.com.html
264 -# Ignore TLS errors
265 -monolith -k https://invalid-cert.site.com -o ~/.local/state/pi/research/pages/site.com.html
267 -# Fallback to curl
268 -curl -sL "https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
269 -```
271 -### curl Failures
273 -```bash
274 -# Retry on failure
275 -until curl -sL "https://example.com" -o ~/.local/state/pi/research/pages/example.com.html; do
276 - echo "Retrying..."
277 - sleep 5
278 -done
280 -# Longer timeout
281 -curl -sL --connect-timeout 60 --max-time 120 "https://slow.site.com" -o page.html
283 -# Ignore invalid SSL
284 -curl -skL "https://invalid-cert.site.com" -o page.html
285 -```
287 -### Git Failures
289 -```bash
290 -# Retry on clone failure
291 -until git clone --depth 1 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo; do
292 - echo "Retrying..."
293 - sleep 5
294 -done
296 -# Specific branch if main fails
297 -git clone --depth 1 --branch develop https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
299 -# Shallow partial clone fallback
300 -git clone --depth 1 --filter=tree:0 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
301 -```
303 -## Pre-flight Check
305 -Before downloading, check what's already available:
307 -```bash
308 -# Check tools available
309 -which monolith # primary web downloader
310 -which curl # fallback
311 -which git # git clones
312 -which rg # ripgrep (optional, faster)
314 -# List existing research
315 -ls ~/.local/state/pi/research/pages/
316 -ls ~/.local/state/pi/research/repos/
317 -```
319 -## Cleanup (Optional)
321 -Only remove if **disk space critical** or **data truly obsolete**.
323 -```bash
324 -# Remove single item
325 -rm -rf ~/.local/state/pi/research/repos/github.com.user.repo
326 -rm ~/.local/state/pi/research/pages/example.com.html
328 -# Remove by age (e.g., older than 30 days)
329 -find ~/.local/state/pi/research -type f -mtime +30 -delete
331 -# Remove empty dirs
332 -find ~/.local/state/pi/research -type d -empty -delete
333 -```
335 -**Default:
336 -do not delete.** Research data persists for reuse.
338 -## Common Issues
340 -### Issue: "mkdir created literal {pages,repos,extracts} dir"
342 -**Fix:** Use separate mkdir commands, not brace expansion:
344 -```bash
345 -mkdir -p ~/.local/state/pi/research/pages
346 -mkdir -p ~/.local/state/pi/research/repos
347 -mkdir -p ~/.local/state/pi/research/extracts
348 -```
350 -### Issue: "monolith fails"
352 -**Fix:** Fallback to curl:
354 -```bash
355 -curl -sL "https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
356 -```
358 -### Issue: "grep finds nothing in HTML"
360 -**Fix:** HTML has lots of noise.
361 -Try ripgrep with `-s` (no warnings):
363 -```bash
364 -rg -s "pattern" ~/.local/state/pi/research/pages/
365 -# Or search for text within specific tags
366 -rg ">([^<]*pattern[^<]*)<" ~/.local/state/pi/research/pages/
367 -```
369 -### Issue: "git clone hangs"
371 -**Fix:** Use timeout and retry:
373 -```bash
374 -timeout 60 git clone --depth 1 URL ~/.local/state/pi/research/repos/name || retry
375 -```
377 -## Pro Tips
379 -- **Use curl for speed**:
380 - It's always available, fast enough for most pages
381 -- **Use monolith for complexity**:
382 - SPAs, pages needing JS rendering, archival
383 -- **Check before clone**:
384 - `ls ~/.local/state/pi/research/repos/` might already have it
385 -- **Shallow first**:
386 - Always `--depth 1` unless history needed
387 -- **Sparse when sure**:
388 - Know exact paths before cloning
389 -- **Name files descriptively**:
390 - Include domain/user/repo for easy grep
391 -- **Never delete by default**:
392 - Assume research data is useful later
393 -- **Use ripgrep if available**:
394 - `rg` is much faster than grep on large files
396 -## Quick Command Reference
398 -```bash
399 -# Setup
400 -mkdir -p ~/.local/state/pi/research/pages
401 -mkdir -p ~/.local/state/pi/research/repos
402 -mkdir -p ~/.local/state/pi/research/extracts
404 -# Check state
405 -ls ~/.local/state/pi/research/pages/
406 -ls ~/.local/state/pi/research/repos/
408 -# Web (monolith - primary)
409 -monolith URL -o ~/.local/state/pi/research/pages/out.html
410 -monolith -a -c -F -i -j -M -I URL -o ~/.local/state/pi/research/pages/out.html # Clean extract
412 -# Web (curl - fallback)
413 -curl -sL URL -o ~/.local/state/pi/research/pages/out.html
415 -# Git
416 -git clone --depth 1 URL ~/.local/state/pi/research/repos/name # Shallow
417 -git sparse-checkout set path path2 # Selective
418 -git clone --depth 1 --filter=tree:0 --sparse URL ~/.local/state/pi/research/repos/name # Both
420 -# Search
421 -rg "pattern" ~/.local/state/pi/research/pages/
422 -rg "pattern" ~/.local/state/pi/research/repos/
423 -rg "pattern" ~/.local/state/pi/research/ # Everything
424 -```
6 +# Research → Wiki
8 +Research has moved to the `wiki` skill.
10 +Use `wiki` for all old research tasks:
12 +| Old research intent | New wiki operation |
13 +| --- | --- |
14 +| Research a topic | `/wiki recall <topic>` then ingest gaps |
15 +| Download/fetch a web page | `/wiki ingest <url>` |
16 +| Fetch docs/code/concepts | `/wiki ingest <url-or-repo>` |
17 +| Clone/search a repository | `/wiki ingest <repo-url>` |
18 +| Save findings for later | `/wiki retro <insight>` or compiled wiki page |
19 +| Search previous research | `/wiki recall <query>` |
21 +Rules:
22 +- Immediately load/use the `wiki` skill for the actual workflow.
23 +- Store new raw evidence under `~/Notes/Wiki/raw-sources/`, not `~/.local/state/pi/research/`.
24 +- Treat existing `~/.local/state/pi/research/{pages,repos,extracts}` and old `~/.pi/research/{pages,repos,extracts}` as legacy cache.
25 +- Search legacy cache before refetching, copy relevant items into `~/Notes/Wiki/raw-sources/legacy-research/`, register them, and never delete by default.

skills/wiki/SKILL.md

Added
new file mode 100644
1 +---
2 +name: wiki
3 +description: "Research and persistent interlinked knowledge base in ~/Notes/Wiki. Use for web downloads, git/repo research, fetching docs/code/concepts, ingesting URLs/files/text, saving or remembering knowledge, recall, synthesis, knowledge base, wiki, 'what do I know about'."
4 +---
6 +# LLM Wiki
8 +Persistent research + knowledge base at **`~/Notes/Wiki/`**. Use it to fetch sources, preserve evidence, distill concepts, and recall prior knowledge. Agent curates; knowledge compounds.
10 +## Layout
12 +```text
13 +~/Notes/Wiki/
14 +├── SCHEMA.md # conventions — read FIRST if it exists
15 +├── index.md # catalog of compiled pages
16 +├── log.md # append-only mutation log
17 +├── raw-sources/ # immutable; append-only evidence
18 +│ ├── index.md # source registry
19 +│ ├── web/ # fetched pages/API responses
20 +│ ├── repos/ # cloned/copied repository snapshots
21 +│ ├── files/ # local files copied for ingestion
22 +│ ├── text/ # pasted/session text
23 +│ ├── archives/ # downloaded/extracted archives
24 +│ └── legacy-research/ # migrated ~/.pi/research items
25 +└── <topic>/ # compiled knowledge pages
26 +```
28 +## Non-negotiable rules
30 +1. **Read `SCHEMA.md` first** if it exists.
31 +2. **Raw sources are immutable.** Never edit after registration; append a new source instead.
32 +3. **Check before fetching.** Search `~/Notes/Wiki/` and legacy `~/.pi/research/` before downloading/cloning.
33 +4. **Local first.** Search registered raw sources before using network.
34 +5. **One compiled file per concept.** Use `kebab-case.md` in the appropriate topic dir.
35 +6. **Cross-reference.** Every compiled page links ≥2 related pages via `[[wikilinks]]` when possible.
36 +7. **Cite sources.** Every factual claim cites raw source IDs or stable source paths.
37 +8. **Keep indexes current.** Update `raw-sources/index.md`, `index.md`, and `log.md` for every mutation.
38 +9. **Never delete by default.** Keep sources for reuse unless user explicitly asks or disk space is critical.
40 +## Frontmatter for compiled pages
42 +```yaml
43 +---
44 +title: Page Title
45 +type: concept | decision | bug | open-question | source | reference | synthesis | stub
46 +updated: YYYY-MM-DD
47 +sources: [raw-sources/source-id-or-path]
48 +tags: [optional]
49 +---
50 +```
52 +## Source IDs and names
54 +Use stable, grep-friendly IDs and filenames:
56 +```text
57 +web/docs.framework.com.guide.2026-05-30.html
58 +web/api.example.com.v2.users.2026-05-30.json
59 +repos/github.com.user.repo.main.2026-05-30/
60 +files/project.path.to.file.2026-05-30.md
61 +text/session-topic.2026-05-30.md
62 +archives/example.com.release.2026-05-30/
63 +legacy-research/github.com.user.repo/
64 +```
66 +Prefer `{host}.{path-or-owner-repo}.{branch-or-topic}.{date}`. Include original URL/path, fetch date, command/tool used, and brief notes in `raw-sources/index.md`.
68 +## Core workflow
70 +```text
71 +1. Read SCHEMA.md if present.
72 +2. Search ~/Notes/Wiki/ and ~/.pi/research/ for existing sources/pages.
73 +3. If missing, fetch/copy/clone into raw-sources/ and register it.
74 +4. Search/read local raw source files.
75 +5. Compile or update concept pages with citations and wikilinks.
76 +6. Update index.md and log.md.
77 +7. Answer with synthesized findings and [[wikilink]] citations.
78 +```
80 +## Operations
82 +### Init (`/wiki init [topic]`)
84 +If `~/Notes/Wiki/` doesn't exist, create:
86 +```bash
87 +mkdir -p ~/Notes/Wiki/raw-sources/web
88 +mkdir -p ~/Notes/Wiki/raw-sources/repos
89 +mkdir -p ~/Notes/Wiki/raw-sources/files
90 +mkdir -p ~/Notes/Wiki/raw-sources/text
91 +mkdir -p ~/Notes/Wiki/raw-sources/archives
92 +mkdir -p ~/Notes/Wiki/raw-sources/legacy-research
93 +```
95 +Then create:
96 +- `raw-sources/index.md` — source registry
97 +- `index.md` — compiled-page catalog
98 +- `log.md` — append-only op log
99 +- `SCHEMA.md` — topic, buckets, taxonomy, citation conventions
101 +### Recall (`/wiki [query]` or `/wiki recall [query]`)
103 +1. Search compiled pages first:
104 + ```bash
105 + rg -i "query terms" ~/Notes/Wiki --glob '!raw-sources/**'
106 + ```
107 +2. Search raw sources if compiled pages are missing/thin:
108 + ```bash
109 + rg -i "query terms" ~/Notes/Wiki/raw-sources ~/.pi/research 2>/dev/null
110 + ```
111 +3. Read matching pages/sources.
112 +4. Synthesize answer with `[[wikilink]]` citations.
113 +5. If the wiki lacks info, say so and suggest exact sources to ingest.
115 +### Ingest (`/wiki ingest <source>`)
117 +Source can be a URL, repository URL, archive URL, file path, directory, pasted text, or legacy `~/.pi/research` item.
119 +1. **Classify** source: web | repo | archive | file | text | legacy.
120 +2. **Check existing** `raw-sources/index.md`, filenames, and `~/.pi/research/` for matching source.
121 +3. **Register** source in `raw-sources/index.md` with stable ID, original location, date, fetch/copy method, and integrity notes.
122 +4. **Capture immutable raw source**:
123 + - Web/API: use `web_fetch` when available for readable Markdown/text; use `monolith` for standalone archival HTML; fallback to `curl -sL` for simple GET/API responses.
124 + - Repos: use `git_clone_safe` when available. Prefer shallow/sparse/partial clones for large repos. Store under `raw-sources/repos/<host.owner.repo...>/`.
125 + - Archives: download to `raw-sources/archives/`, extract into a dated subdir, keep original archive when practical.
126 + - Files/dirs: copy into `raw-sources/files/`; do not mutate the copy.
127 + - Text: save pasted/session text under `raw-sources/text/`.
128 + - Legacy: copy relevant `~/.pi/research/{pages,repos,extracts}` item into `raw-sources/legacy-research/`; do not delete legacy data.
129 +5. **Inspect locally** with `rg`, `find`, `read`, etc. before more network calls.
130 +6. **Compile** into focused pages under `<topic>/`. Update existing pages when concepts overlap.
131 +7. **Cross-link** related pages and cite raw source IDs.
132 +8. **Index/log** updates.
134 +### Retro (`/wiki retro [insight]`)
136 +Save an atomic lesson from the current session:
137 +1. Create/update one concept/decision/bug page.
138 +2. Cite `raw-sources/text/<session-or-note-id>` if the lesson comes from conversation context.
139 +3. Add ≥2 wikilinks when relevant.
140 +4. Update `index.md` and `log.md`.
142 +### Lint (`/wiki lint`)
144 +Health check:
145 +- `index.md` ↔ filesystem sync: missing pages, orphan entries
146 +- `raw-sources/index.md` ↔ raw files sync
147 +- broken wikilinks
148 +- compiled pages without citations
149 +- stub aging >30 days
150 +- orphan pages without inbound links
151 +- likely contradictions across pages
153 +Fix deterministic issues. Report heuristic issues.
155 +### Status (`/wiki status`)
157 +Report page count by type, source count by bucket, last activity, stale stubs, orphan pages, and notable gaps.
159 +### Digest (`/wiki digest [daily|weekly]`)
161 +Summarize recent activity from `log.md`: sources added, pages changed, decisions, open questions.
163 +## Fetch guidance
165 +| Need | Preferred | Fallback |
166 +| --- | --- | --- |
167 +| Read a normal URL | `web_fetch` | `curl -sL` |
168 +| Archive complete web page | `monolith -q URL -o raw-sources/web/name.html` | `curl -sL URL -o ...` |
169 +| Clean readable HTML snapshot | `monolith -a -c -F -i -j -M -I URL -o ...` | `web_fetch` markdown/text |
170 +| API JSON/text | `web_fetch` or `curl -sL` | browser/tool-specific fetch |
171 +| External repo | `git_clone_safe URL raw-sources/repos/name` | user-approved safe shallow/sparse clone |
172 +| Large repo | safe shallow/sparse/partial clone | ingest only relevant files/docs |
173 +| Existing old research | copy from `~/.pi/research/` | cite legacy path if copying is too large |
175 +## Search patterns
177 +```bash
178 +# Existing wiki knowledge
179 +rg -i "pattern" ~/Notes/Wiki --glob '!raw-sources/**'
181 +# Registered sources
182 +rg -i "pattern" ~/Notes/Wiki/raw-sources/
184 +# Legacy research cache
185 +rg -i "pattern" ~/.pi/research/ 2>/dev/null
187 +# Repo source with context
188 +rg -n -A2 -B2 "function_name|concept" ~/Notes/Wiki/raw-sources/repos/<repo>/
190 +# HTML text snippets when markup is noisy
191 +rg -s ">([^<]*pattern[^<]*)<" ~/Notes/Wiki/raw-sources/web/
192 +```
194 +## Legacy research migration
196 +Old research lived in `~/.pi/research/{pages,repos,extracts}`. It is now a cache of historical raw sources.
198 +When relevant:
199 +1. Search it before fetching new data.
200 +2. Copy useful items into `~/Notes/Wiki/raw-sources/legacy-research/` or cite the legacy path if too large.
201 +3. Register copied/cited items in `raw-sources/index.md`.
202 +4. Compile durable concepts into normal topic pages.
203 +5. Do **not** delete `~/.pi/research/` by default.
205 +## Wikilink convention
207 +`[[topic/page-name]]` resolves to `~/Notes/Wiki/topic/page-name.md`.
208 +In portable Markdown, also acceptable: `[page name](../topic/page-name.md)`.
210 +## Quality bar
212 +- `concept` pages: clear summary, evidence, implications, related pages.
213 +- `reference` pages: commands/API details copied from cited source, with version/date.
214 +- `synthesis` pages: cite ≥2 sources or compiled pages.
215 +- `stub` pages: frontmatter + rationale + see-also; lint flags stubs older than 30 days.
216 +- Answers distinguish sourced facts from inference.