13 -Research data lives in `~/.pi/research/` — **never delete after use**.
13 +Research data lives in `~/.local/state/pi/research/` — **never delete after use**.
16 +~/.local/state/pi/research/
17 17 ├── pages/ # web page downloads (HTML)
18 18 ├── repos/ # git clones
19 19 └── extracts/ # extracted archives
26 -mkdir -p ~/.pi/research/pages
27 -mkdir -p ~/.pi/research/repos
28 -mkdir -p ~/.pi/research/extracts
26 +mkdir -p ~/.local/state/pi/research/pages
27 +mkdir -p ~/.local/state/pi/research/repos
28 +mkdir -p ~/.local/state/pi/research/extracts
37 -1. Check ~/.pi/research/ — maybe already downloaded
37 +1. Check ~/.local/state/pi/research/ — maybe already downloaded
38 38 2. If not → download/clone to appropriate subfolder
39 39 3. Search within local files first
51 -monolith https://example.com -o ~/.pi/research/pages/example.com.html
51 +monolith https://example.com -o ~/.local/state/pi/research/pages/example.com.html
54 -monolith -a -c -F -i -j -M -I https://example.com -o ~/.pi/research/pages/example.com.clean.html
54 +monolith -a -c -F -i -j -M -I https://example.com -o ~/.local/state/pi/research/pages/example.com.clean.html
57 -monolith -q https://example.com -o ~/.pi/research/pages/example.com.html
57 +monolith -q https://example.com -o ~/.local/state/pi/research/pages/example.com.html
60 -monolith -t 120 https://example.com -o ~/.pi/research/pages/example.com.html
60 +monolith -t 120 https://example.com -o ~/.local/state/pi/research/pages/example.com.html
69 -curl -sL
"https://example.com" -o ~/.pi/research/pages/example.com.html
69 +curl -sL
"https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
72 -curl -sL
"https://docs.framework.com/guide" -o ~/.pi/research/pages/docs.framework.com.guide.html
72 +curl -sL
"https://docs.framework.com/guide" -o ~/.local/state/pi/research/pages/docs.framework.com.guide.html
112 112 ###
Shallow Clone (single branch, history truncated) 115 -git clone --depth 1 https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
115 +git clone --depth 1 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
118 118 ###
Shallow Clone Specific Branch 121 -git clone --depth 1 --branch main https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
121 +git clone --depth 1 --branch main https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
124 124 ###
Sparse Clone (only wanted directories) 128 -git init ~/.pi/research/repos/github.com.user.repo
129 -cd ~/.pi/research/repos/github.com.user.repo
128 +git init ~/.local/state/pi/research/repos/github.com.user.repo
129 +cd ~/.local/state/pi/research/repos/github.com.user.repo
132 132 git remote add origin https://github.com/user/repo
144 144 ###
Sparse + Shallow Combined 147 -git clone --depth 1 --filter
=blob:none --sparse https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
148 -cd ~/.pi/research/repos/github.com.user.repo
147 +git clone --depth 1 --filter
=blob:none --sparse https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
148 +cd ~/.local/state/pi/research/repos/github.com.user.repo
149 149 git sparse-checkout set src docs
150 150 git checkout main
153 153 ###
Partial Clone (fetch only reachable objects) 156 -git clone --depth 1 --filter
=tree:0 https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
156 +git clone --depth 1 --filter
=tree:0 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
159 159 ###
Repo Naming Convention 173 -ls ~/.pi/research/pages/
173 +ls ~/.local/state/pi/research/pages/
176 -ls ~/.pi/research/repos/
176 +ls ~/.local/state/pi/research/repos/
179 -ls ~/.pi/research/pages/
| grep domain
180 -ls ~/.pi/research/repos/
| grep user
179 +ls ~/.local/state/pi/research/pages/
| grep domain
180 +ls ~/.local/state/pi/research/repos/
| grep user
183 183 ###
Within Downloaded Pages 187 -grep -n
"pattern" ~/.pi/research/pages/
*.html
187 +grep -n
"pattern" ~/.local/state/pi/research/pages/
*.html
190 -grep -rn
"pattern" ~/.pi/research/pages/
190 +grep -rn
"pattern" ~/.local/state/pi/research/pages/
193 -grep -ri
"pattern" ~/.pi/research/pages/
193 +grep -ri
"pattern" ~/.local/state/pi/research/pages/
196 -rg
"pattern" ~/.pi/research/pages/
196 +rg
"pattern" ~/.local/state/pi/research/pages/
199 199 ###
Within Cloned Repos 203 -grep -rn
"function_name" ~/.pi/research/repos/github.com.user.repo/src/
203 +grep -rn
"function_name" ~/.local/state/pi/research/repos/github.com.user.repo/src/
206 -find ~/.pi/research/repos/github.com.user.repo -name
"*.py" -exec grep -l
"pattern" {}
\; 206 +find ~/.local/state/pi/research/repos/github.com.user.repo -name
"*.py" -exec grep -l
"pattern" {}
\; 209 -grep -rn -A2 -B2
"pattern" ~/.pi/research/repos/github.com.user.repo/
209 +grep -rn -A2 -B2
"pattern" ~/.local/state/pi/research/repos/github.com.user.repo/
212 -rg -A2 -B2
"pattern" ~/.pi/research/repos/github.com.user.repo/
212 +rg -A2 -B2
"pattern" ~/.local/state/pi/research/repos/github.com.user.repo/
215 215 ###
Search Git History (if full clone needed) 218 -cd ~/.pi/research/repos/github.com.user.repo
218 +cd ~/.local/state/pi/research/repos/github.com.user.repo
220 220 git log --oneline -20
221 221 git log --all --oneline --graph
229 -rg
"pattern" ~/.pi/research/pages/
229 +rg
"pattern" ~/.local/state/pi/research/pages/
232 -rg
"pattern" ~/.pi/research/repos/
232 +rg
"pattern" ~/.local/state/pi/research/repos/
235 -rg
"pattern" ~/.pi/research/
235 +rg
"pattern" ~/.local/state/pi/research/
238 238 ##
Extract Archives 242 -curl -sL https://example.com/archive.tar.gz
| tar -xz -C ~/.pi/research/extracts/
242 +curl -sL https://example.com/archive.tar.gz
| tar -xz -C ~/.local/state/pi/research/extracts/
245 245 curl -sL https://example.com/archive.zip -o /tmp/archive.zip
246 -unzip /tmp/archive.zip -d ~/.pi/research/extracts/
246 +unzip /tmp/archive.zip -d ~/.local/state/pi/research/extracts/
249 -wget -O ~/.pi/research/extracts/archive.tar.gz https://example.com/archive.tar.gz
250 -tar -xzf ~/.pi/research/extracts/archive.tar.gz -C ~/.pi/research/extracts/
249 +wget -O ~/.local/state/pi/research/extracts/archive.tar.gz https://example.com/archive.tar.gz
250 +tar -xzf ~/.local/state/pi/research/extracts/archive.tar.gz -C ~/.local/state/pi/research/extracts/
253 253 ##
Error Handling 259 -monolith -t 120 https://slow.site.com -o ~/.pi/research/pages/site.com.html
259 +monolith -t 120 https://slow.site.com -o ~/.local/state/pi/research/pages/site.com.html
262 -monolith -e https://maybe-down.site.com -o ~/.pi/research/pages/site.com.html
262 +monolith -e https://maybe-down.site.com -o ~/.local/state/pi/research/pages/site.com.html
265 -monolith -k https://invalid-cert.site.com -o ~/.pi/research/pages/site.com.html
265 +monolith -k https://invalid-cert.site.com -o ~/.local/state/pi/research/pages/site.com.html
268 -curl -sL
"https://example.com" -o ~/.pi/research/pages/example.com.html
268 +curl -sL
"https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
271 271 ###
curl Failures 275 -until curl -sL
"https://example.com" -o ~/.pi/research/pages/example.com.html
; do 275 +until curl -sL
"https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
; do 276 276 echo "Retrying..." 291 -until git clone --depth 1 https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
; do 291 +until git clone --depth 1 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
; do 292 292 echo "Retrying..." 297 -git clone --depth 1 --branch develop https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
297 +git clone --depth 1 --branch develop https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
300 -git clone --depth 1 --filter
=tree:0 https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
300 +git clone --depth 1 --filter
=tree:0 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
303 303 ##
Pre-flight Check 312 312 which rg # ripgrep (optional, faster)
314 314 #
List existing research 315 -ls ~/.pi/research/pages/
316 -ls ~/.pi/research/repos/
315 +ls ~/.local/state/pi/research/pages/
316 +ls ~/.local/state/pi/research/repos/
319 319 ## Cleanup (Optional)
325 -rm -rf ~/.pi/research/repos/github.com.user.repo
326 -rm ~/.pi/research/pages/example.com.html
325 +rm -rf ~/.local/state/pi/research/repos/github.com.user.repo
326 +rm ~/.local/state/pi/research/pages/example.com.html
329 -find ~/.pi/research -type f -mtime +30 -delete
329 +find ~/.local/state/pi/research -type f -mtime +30 -delete
332 -find ~/.pi/research -type d -empty -delete
332 +find ~/.local/state/pi/research -type d -empty -delete
342 342 **Fix:** Use separate mkdir commands, not brace expansion:
345 -mkdir -p ~/.pi/research/pages
346 -mkdir -p ~/.pi/research/repos
347 -mkdir -p ~/.pi/research/extracts
345 +mkdir -p ~/.local/state/pi/research/pages
346 +mkdir -p ~/.local/state/pi/research/repos
347 +mkdir -p ~/.local/state/pi/research/extracts
350 350 ###
Issue: "monolith fails" 352 352 **Fix:** Fallback to curl:
355 -curl -sL
"https://example.com" -o ~/.pi/research/pages/example.com.html
355 +curl -sL
"https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
358 358 ###
Issue: "grep finds nothing in HTML" 361 361 Try ripgrep with `-s` (no warnings):
364 -rg -s
"pattern" ~/.pi/research/pages/
364 +rg -s
"pattern" ~/.local/state/pi/research/pages/
366 -rg
">([^<]*pattern[^<]*)<" ~/.pi/research/pages/
366 +rg
">([^<]*pattern[^<]*)<" ~/.local/state/pi/research/pages/
369 369 ###
Issue: "git clone hangs" 371 371 **Fix:** Use timeout and retry:
374 -timeout 60 git clone --depth 1 URL ~/.pi/research/repos/name
|| retry
374 +timeout 60 git clone --depth 1 URL ~/.local/state/pi/research/repos/name
|| retry
381 381 - **Use monolith for complexity**:
382 382 SPAs, pages needing JS rendering, archival
383 383 - **Check before clone**:
384 - `ls ~/.pi/research/repos/` might already have it
384 + `ls ~/.local/state/pi/research/repos/` might already have it
385 385 - **Shallow first**:
386 386 Always `--depth 1` unless history needed
387 387 - **Sparse when sure**:
400 -mkdir -p ~/.pi/research/pages
401 -mkdir -p ~/.pi/research/repos
402 -mkdir -p ~/.pi/research/extracts
400 +mkdir -p ~/.local/state/pi/research/pages
401 +mkdir -p ~/.local/state/pi/research/repos
402 +mkdir -p ~/.local/state/pi/research/extracts
405 -ls ~/.pi/research/pages/
406 -ls ~/.pi/research/repos/
405 +ls ~/.local/state/pi/research/pages/
406 +ls ~/.local/state/pi/research/repos/
409 -monolith URL -o ~/.pi/research/pages/out.html
410 -monolith -a -c -F -i -j -M -I URL -o ~/.pi/research/pages/out.html
409 +monolith URL -o ~/.local/state/pi/research/pages/out.html
410 +monolith -a -c -F -i -j -M -I URL -o ~/.local/state/pi/research/pages/out.html
413 -curl -sL URL -o ~/.pi/research/pages/out.html
413 +curl -sL URL -o ~/.local/state/pi/research/pages/out.html
416 -git clone --depth 1 URL ~/.pi/research/repos/name
416 +git clone --depth 1 URL ~/.local/state/pi/research/repos/name
417 417 git sparse-checkout set path path2
418 -git clone --depth 1 --filter
=tree:0 --sparse URL ~/.pi/research/repos/name
418 +git clone --depth 1 --filter
=tree:0 --sparse URL ~/.local/state/pi/research/repos/name
421 -rg
"pattern" ~/.pi/research/pages/
422 -rg
"pattern" ~/.pi/research/repos/
423 -rg
"pattern" ~/.pi/research/
421 +rg
"pattern" ~/.local/state/pi/research/pages/
422 +rg
"pattern" ~/.local/state/pi/research/repos/
423 +rg
"pattern" ~/.local/state/pi/research/