3 -description: Research via web downloads and git clones. Fetch docs, code, concepts. Use curl or monolith for pages, git sparse/shallow for repos. Persisted to state folder.
3 +description: "Compatibility alias for wiki research. Trigger on research, web downloads, git/repo research, fetching docs/code/concepts, curl/monolith pages, sparse/shallow repos, persisted research state. Use wiki instead."
8 -Web and repository research.
9 -Download, clone, extract, persist.
13 -Research data lives in `~/.local/state/pi/research/` — **never delete after use**.
16 -~/.local/state/pi/research/
17 -├── pages/ # web page downloads (HTML)
18 -├── repos/ # git clones
19 -└── extracts/ # extracted archives
22 -###
Create Structure (Important: Use Correct Syntax) 26 -mkdir -p ~/.local/state/pi/research/pages
27 -mkdir -p ~/.local/state/pi/research/repos
28 -mkdir -p ~/.local/state/pi/research/extracts
37 -1. Check ~/.local/state/pi/research/ — maybe already downloaded
38 -2. If not → download/clone to appropriate subfolder
39 -3. Search within local files first
43 -##
Downloading Web Pages 45 -###
Primary: monolith (Recommended) 47 -monolith saves pages as standalone HTML — no external deps, works offline.
51 -monolith https://example.com -o ~/.local/state/pi/research/pages/example.com.html
54 -monolith -a -c -F -i -j -M -I https://example.com -o ~/.local/state/pi/research/pages/example.com.clean.html
57 -monolith -q https://example.com -o ~/.local/state/pi/research/pages/example.com.html
60 -monolith -t 120 https://example.com -o ~/.local/state/pi/research/pages/example.com.html
65 -Use when monolith fails or for simple GET requests.
69 -curl -sL
"https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
72 -curl -sL
"https://docs.framework.com/guide" -o ~/.local/state/pi/research/pages/docs.framework.com.guide.html
77 -- `-s` = silent (no progress meter)
78 -- `-L` = follow redirects
79 -- `-o file` = output to file
81 -**monolith flags for clean extract:**
87 -- `-j` = no JavaScript
89 -- `-I` = isolate (no external refs)
91 -###
Which Method to Use? 94 -| ----------------------------------- | ------------------------------- |
95 -| Most pages | `monolith` (standalone HTML) |
96 -| Complex page (SPA, dynamic content) | `monolith` |
97 -| Readability focused | `monolith -a -c -F -i -j -M -I` |
98 -| Quick text fetch | `curl` |
99 -| API response (JSON) | `curl` |
101 -###
Naming Convention 104 -{domain}.{path}.{suffix}.html
105 -example.com.docs.html
106 -github.com.user.repo.readme.html
107 -docs.framework.com.api.v2.html
112 -###
Shallow Clone (single branch, history truncated) 115 -git clone --depth 1 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
118 -###
Shallow Clone Specific Branch 121 -git clone --depth 1 --branch main https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
124 -###
Sparse Clone (only wanted directories) 128 -git init ~/.local/state/pi/research/repos/github.com.user.repo
129 -cd ~/.local/state/pi/research/repos/github.com.user.repo
132 -git remote add origin https://github.com/user/repo
135 -git sparse-checkout init --cone
138 -git sparse-checkout set src/docs
141 -git pull origin main
144 -###
Sparse + Shallow Combined 147 -git clone --depth 1 --filter
=blob:none --sparse https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
148 -cd ~/.local/state/pi/research/repos/github.com.user.repo
149 -git sparse-checkout set src docs
153 -###
Partial Clone (fetch only reachable objects) 156 -git clone --depth 1 --filter
=tree:0 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
159 -###
Repo Naming Convention 162 -{host}.{user}.{repo}[.{branch}][.{path}]
163 -github.com.user.library.main.html
164 -github.com.user.framework.src.html
169 -###
Check Before Download 173 -ls ~/.local/state/pi/research/pages/
176 -ls ~/.local/state/pi/research/repos/
179 -ls ~/.local/state/pi/research/pages/
| grep domain
180 -ls ~/.local/state/pi/research/repos/
| grep user
183 -###
Within Downloaded Pages 187 -grep -n
"pattern" ~/.local/state/pi/research/pages/
*.html
190 -grep -rn
"pattern" ~/.local/state/pi/research/pages/
193 -grep -ri
"pattern" ~/.local/state/pi/research/pages/
196 -rg
"pattern" ~/.local/state/pi/research/pages/
199 -###
Within Cloned Repos 203 -grep -rn
"function_name" ~/.local/state/pi/research/repos/github.com.user.repo/src/
206 -find ~/.local/state/pi/research/repos/github.com.user.repo -name
"*.py" -exec grep -l
"pattern" {}
\; 209 -grep -rn -A2 -B2
"pattern" ~/.local/state/pi/research/repos/github.com.user.repo/
212 -rg -A2 -B2
"pattern" ~/.local/state/pi/research/repos/github.com.user.repo/
215 -###
Search Git History (if full clone needed) 218 -cd ~/.local/state/pi/research/repos/github.com.user.repo
220 -git log --oneline -20
221 -git log --all --oneline --graph
222 -git log -p --follow
-- filename
225 -###
Across Everything 229 -rg
"pattern" ~/.local/state/pi/research/pages/
232 -rg
"pattern" ~/.local/state/pi/research/repos/
235 -rg
"pattern" ~/.local/state/pi/research/
242 -curl -sL https://example.com/archive.tar.gz
| tar -xz -C ~/.local/state/pi/research/extracts/
245 -curl -sL https://example.com/archive.zip -o /tmp/archive.zip
246 -unzip /tmp/archive.zip -d ~/.local/state/pi/research/extracts/
249 -wget -O ~/.local/state/pi/research/extracts/archive.tar.gz https://example.com/archive.tar.gz
250 -tar -xzf ~/.local/state/pi/research/extracts/archive.tar.gz -C ~/.local/state/pi/research/extracts/
255 -###
monolith Failures 259 -monolith -t 120 https://slow.site.com -o ~/.local/state/pi/research/pages/site.com.html
262 -monolith -e https://maybe-down.site.com -o ~/.local/state/pi/research/pages/site.com.html
265 -monolith -k https://invalid-cert.site.com -o ~/.local/state/pi/research/pages/site.com.html
268 -curl -sL
"https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
275 -until curl -sL
"https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
; do 281 -curl -sL --connect-timeout 60 --max-time 120
"https://slow.site.com" -o page.html
284 -curl -skL
"https://invalid-cert.site.com" -o page.html
291 -until git clone --depth 1 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
; do 297 -git clone --depth 1 --branch develop https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
300 -git clone --depth 1 --filter
=tree:0 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
305 -Before downloading, check what's already available:
315 -ls ~/.local/state/pi/research/pages/
316 -ls ~/.local/state/pi/research/repos/
319 -##
Cleanup (Optional) 321 -Only remove if **disk space critical** or **data truly obsolete**.
325 -rm -rf ~/.local/state/pi/research/repos/github.com.user.repo
326 -rm ~/.local/state/pi/research/pages/example.com.html
329 -find ~/.local/state/pi/research -type f -mtime +30 -delete
332 -find ~/.local/state/pi/research -type d -empty -delete
336 -do not delete.** Research data persists for reuse.
340 -###
Issue: "mkdir created literal {pages,repos,extracts} dir" 342 -**Fix:** Use separate mkdir commands, not brace expansion:
345 -mkdir -p ~/.local/state/pi/research/pages
346 -mkdir -p ~/.local/state/pi/research/repos
347 -mkdir -p ~/.local/state/pi/research/extracts
350 -###
Issue: "monolith fails" 352 -**Fix:** Fallback to curl:
355 -curl -sL
"https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
358 -###
Issue: "grep finds nothing in HTML" 360 -**Fix:** HTML has lots of noise.
361 -Try ripgrep with `-s` (no warnings):
364 -rg -s
"pattern" ~/.local/state/pi/research/pages/
366 -rg
">([^<]*pattern[^<]*)<" ~/.local/state/pi/research/pages/
369 -###
Issue: "git clone hangs" 371 -**Fix:** Use timeout and retry:
374 -timeout 60 git clone --depth 1 URL ~/.local/state/pi/research/repos/name
|| retry
379 -- **Use curl for speed**:
380 - It's always available, fast enough for most pages
381 -- **Use monolith for complexity**:
382 - SPAs, pages needing JS rendering, archival
383 -- **Check before clone**:
384 - `ls ~/.local/state/pi/research/repos/` might already have it
385 -- **Shallow first**:
386 - Always `--depth 1` unless history needed
387 -- **Sparse when sure**:
388 - Know exact paths before cloning
389 -- **Name files descriptively**:
390 - Include domain/user/repo for easy grep
391 -- **Never delete by default**:
392 - Assume research data is useful later
393 -- **Use ripgrep if available**:
394 - `rg` is much faster than grep on large files
396 -##
Quick Command Reference 400 -mkdir -p ~/.local/state/pi/research/pages
401 -mkdir -p ~/.local/state/pi/research/repos
402 -mkdir -p ~/.local/state/pi/research/extracts
405 -ls ~/.local/state/pi/research/pages/
406 -ls ~/.local/state/pi/research/repos/
409 -monolith URL -o ~/.local/state/pi/research/pages/out.html
410 -monolith -a -c -F -i -j -M -I URL -o ~/.local/state/pi/research/pages/out.html
413 -curl -sL URL -o ~/.local/state/pi/research/pages/out.html
416 -git clone --depth 1 URL ~/.local/state/pi/research/repos/name
417 -git sparse-checkout set path path2
418 -git clone --depth 1 --filter
=tree:0 --sparse URL ~/.local/state/pi/research/repos/name
421 -rg
"pattern" ~/.local/state/pi/research/pages/
422 -rg
"pattern" ~/.local/state/pi/research/repos/
423 -rg
"pattern" ~/.local/state/pi/research/
8 +Research has moved to the `wiki` skill.
10 +Use `wiki` for all old research tasks:
12 +| Old research intent | New wiki operation |
14 +| Research a topic | `/wiki recall <topic>` then ingest gaps |
15 +| Download/fetch a web page | `/wiki ingest <url>` |
16 +| Fetch docs/code/concepts | `/wiki ingest <url-or-repo>` |
17 +| Clone/search a repository | `/wiki ingest <repo-url>` |
18 +| Save findings for later | `/wiki retro <insight>` or compiled wiki page |
19 +| Search previous research | `/wiki recall <query>` |
22 +- Immediately load/use the `wiki` skill for the actual workflow.
23 +- Store new raw evidence under `~/Notes/Wiki/raw-sources/`, not `~/.local/state/pi/research/`.
24 +- Treat existing `~/.local/state/pi/research/{pages,repos,extracts}` and old `~/.pi/research/{pages,repos,extracts}` as legacy cache.
25 +- Search legacy cache before refetching, copy relevant items into `~/Notes/Wiki/raw-sources/legacy-research/`, register them, and never delete by default.