repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

docs(research): update state directory

946a4330ef7cdf5998ff1302de3d1762974bbb11

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (1)

skills/research/SKILL.md

Modified
11 11 ## State Folder
13 -Research data lives in `~/.pi/research/` — **never delete after use**.
13 +Research data lives in `~/.local/state/pi/research/` — **never delete after use**.
15 15 ```text
16 -~/.pi/research/
16 +~/.local/state/pi/research/
17 17 ├── pages/ # web page downloads (HTML)
18 18 ├── repos/ # git clones
19 19 └── extracts/ # extracted archives
24 24 ```bash
25 25 # CORRECT: separate mkdir commands
26 -mkdir -p ~/.pi/research/pages
27 -mkdir -p ~/.pi/research/repos
28 -mkdir -p ~/.pi/research/extracts
26 +mkdir -p ~/.local/state/pi/research/pages
27 +mkdir -p ~/.local/state/pi/research/repos
28 +mkdir -p ~/.local/state/pi/research/extracts
30 30 # WRONG: brace expansion may create literal dir named "{pages,repos,extracts}"
31 -# mkdir -p ~/.pi/research/{pages,repos,extracts} # AVOID THIS
31 +# mkdir -p ~/.local/state/pi/research/{pages,repos,extracts} # AVOID THIS
34 34 ## Core Workflow
36 36 ```text
37 -1. Check ~/.pi/research/ — maybe already downloaded
37 +1. Check ~/.local/state/pi/research/ — maybe already downloaded
38 38 2. If not → download/clone to appropriate subfolder
39 39 3. Search within local files first
40 40 4. Return findings
49 49 ```bash
50 50 # Basic
51 -monolith https://example.com -o ~/.pi/research/pages/example.com.html
51 +monolith https://example.com -o ~/.local/state/pi/research/pages/example.com.html
53 53 # Clean extract (remove JS, CSS, fonts, images, audio, video, metadata)
54 -monolith -a -c -F -i -j -M -I https://example.com -o ~/.pi/research/pages/example.com.clean.html
54 +monolith -a -c -F -i -j -M -I https://example.com -o ~/.local/state/pi/research/pages/example.com.clean.html
56 56 # Quiet mode (less output)
57 -monolith -q https://example.com -o ~/.pi/research/pages/example.com.html
57 +monolith -q https://example.com -o ~/.local/state/pi/research/pages/example.com.html
59 59 # Custom timeout (for slow sites)
60 -monolith -t 120 https://example.com -o ~/.pi/research/pages/example.com.html
60 +monolith -t 120 https://example.com -o ~/.local/state/pi/research/pages/example.com.html
63 63 ### Fallback: curl
67 67 ```bash
68 68 # Basic download
69 -curl -sL "https://example.com" -o ~/.pi/research/pages/example.com.html
69 +curl -sL "https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
71 71 # With better naming (domain + path)
72 -curl -sL "https://docs.framework.com/guide" -o ~/.pi/research/pages/docs.framework.com.guide.html
72 +curl -sL "https://docs.framework.com/guide" -o ~/.local/state/pi/research/pages/docs.framework.com.guide.html
75 75 **curl flags:**
112 112 ### Shallow Clone (single branch, history truncated)
114 114 ```bash
115 -git clone --depth 1 https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
115 +git clone --depth 1 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
118 118 ### Shallow Clone Specific Branch
120 120 ```bash
121 -git clone --depth 1 --branch main https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
121 +git clone --depth 1 --branch main https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
124 124 ### Sparse Clone (only wanted directories)
126 126 ```bash
127 127 # Step 1: Init
128 -git init ~/.pi/research/repos/github.com.user.repo
129 -cd ~/.pi/research/repos/github.com.user.repo
128 +git init ~/.local/state/pi/research/repos/github.com.user.repo
129 +cd ~/.local/state/pi/research/repos/github.com.user.repo
131 131 # Step 2: Add remote
132 132 git remote add origin https://github.com/user/repo
144 144 ### Sparse + Shallow Combined
146 146 ```bash
147 -git clone --depth 1 --filter=blob:none --sparse https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
148 -cd ~/.pi/research/repos/github.com.user.repo
147 +git clone --depth 1 --filter=blob:none --sparse https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
148 +cd ~/.local/state/pi/research/repos/github.com.user.repo
149 149 git sparse-checkout set src docs
150 150 git checkout main
153 153 ### Partial Clone (fetch only reachable objects)
155 155 ```bash
156 -git clone --depth 1 --filter=tree:0 https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
156 +git clone --depth 1 --filter=tree:0 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
159 159 ### Repo Naming Convention
171 171 ```bash
172 172 # List already downloaded pages
173 -ls ~/.pi/research/pages/
173 +ls ~/.local/state/pi/research/pages/
175 175 # List already cloned repos
176 -ls ~/.pi/research/repos/
176 +ls ~/.local/state/pi/research/repos/
178 178 # Find by name (grep)
179 -ls ~/.pi/research/pages/ | grep domain
180 -ls ~/.pi/research/repos/ | grep user
179 +ls ~/.local/state/pi/research/pages/ | grep domain
180 +ls ~/.local/state/pi/research/repos/ | grep user
183 183 ### Within Downloaded Pages
185 185 ```bash
186 186 # Find pattern in single page
187 -grep -n "pattern" ~/.pi/research/pages/*.html
187 +grep -n "pattern" ~/.local/state/pi/research/pages/*.html
189 189 # Find across all pages
190 -grep -rn "pattern" ~/.pi/research/pages/
190 +grep -rn "pattern" ~/.local/state/pi/research/pages/
192 192 # Case-insensitive
193 -grep -ri "pattern" ~/.pi/research/pages/
193 +grep -ri "pattern" ~/.local/state/pi/research/pages/
195 195 # With ripgrep (faster, if available)
196 -rg "pattern" ~/.pi/research/pages/
196 +rg "pattern" ~/.local/state/pi/research/pages/
199 199 ### Within Cloned Repos
201 201 ```bash
202 202 # Find in specific files
203 -grep -rn "function_name" ~/.pi/research/repos/github.com.user.repo/src/
203 +grep -rn "function_name" ~/.local/state/pi/research/repos/github.com.user.repo/src/
205 205 # Find by file extension
206 -find ~/.pi/research/repos/github.com.user.repo -name "*.py" -exec grep -l "pattern" {} \;
206 +find ~/.local/state/pi/research/repos/github.com.user.repo -name "*.py" -exec grep -l "pattern" {} \;
208 208 # Find and show context
209 -grep -rn -A2 -B2 "pattern" ~/.pi/research/repos/github.com.user.repo/
209 +grep -rn -A2 -B2 "pattern" ~/.local/state/pi/research/repos/github.com.user.repo/
211 211 # With ripgrep
212 -rg -A2 -B2 "pattern" ~/.pi/research/repos/github.com.user.repo/
212 +rg -A2 -B2 "pattern" ~/.local/state/pi/research/repos/github.com.user.repo/
215 215 ### Search Git History (if full clone needed)
217 217 ```bash
218 -cd ~/.pi/research/repos/github.com.user.repo
218 +cd ~/.local/state/pi/research/repos/github.com.user.repo
220 220 git log --oneline -20
221 221 git log --all --oneline --graph
227 227 ```bash
228 228 # All pages
229 -rg "pattern" ~/.pi/research/pages/
229 +rg "pattern" ~/.local/state/pi/research/pages/
231 231 # All repos
232 -rg "pattern" ~/.pi/research/repos/
232 +rg "pattern" ~/.local/state/pi/research/repos/
234 234 # Everything
235 -rg "pattern" ~/.pi/research/
235 +rg "pattern" ~/.local/state/pi/research/
238 238 ## Extract Archives
240 240 ```bash
241 241 # Download and extract tar.gz
242 -curl -sL https://example.com/archive.tar.gz | tar -xz -C ~/.pi/research/extracts/
242 +curl -sL https://example.com/archive.tar.gz | tar -xz -C ~/.local/state/pi/research/extracts/
244 244 # Download and extract zip
245 245 curl -sL https://example.com/archive.zip -o /tmp/archive.zip
246 -unzip /tmp/archive.zip -d ~/.pi/research/extracts/
246 +unzip /tmp/archive.zip -d ~/.local/state/pi/research/extracts/
248 248 # Or download first
249 -wget -O ~/.pi/research/extracts/archive.tar.gz https://example.com/archive.tar.gz
250 -tar -xzf ~/.pi/research/extracts/archive.tar.gz -C ~/.pi/research/extracts/
249 +wget -O ~/.local/state/pi/research/extracts/archive.tar.gz https://example.com/archive.tar.gz
250 +tar -xzf ~/.local/state/pi/research/extracts/archive.tar.gz -C ~/.local/state/pi/research/extracts/
253 253 ## Error Handling
257 257 ```bash
258 258 # Longer timeout
259 -monolith -t 120 https://slow.site.com -o ~/.pi/research/pages/site.com.html
259 +monolith -t 120 https://slow.site.com -o ~/.local/state/pi/research/pages/site.com.html
261 261 # Ignore network errors
262 -monolith -e https://maybe-down.site.com -o ~/.pi/research/pages/site.com.html
262 +monolith -e https://maybe-down.site.com -o ~/.local/state/pi/research/pages/site.com.html
264 264 # Ignore TLS errors
265 -monolith -k https://invalid-cert.site.com -o ~/.pi/research/pages/site.com.html
265 +monolith -k https://invalid-cert.site.com -o ~/.local/state/pi/research/pages/site.com.html
267 267 # Fallback to curl
268 -curl -sL "https://example.com" -o ~/.pi/research/pages/example.com.html
268 +curl -sL "https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
271 271 ### curl Failures
273 273 ```bash
274 274 # Retry on failure
275 -until curl -sL "https://example.com" -o ~/.pi/research/pages/example.com.html; do
275 +until curl -sL "https://example.com" -o ~/.local/state/pi/research/pages/example.com.html; do
276 276 echo "Retrying..."
277 277 sleep 5
289 289 ```bash
290 290 # Retry on clone failure
291 -until git clone --depth 1 https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo; do
291 +until git clone --depth 1 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo; do
292 292 echo "Retrying..."
293 293 sleep 5
296 296 # Specific branch if main fails
297 -git clone --depth 1 --branch develop https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
297 +git clone --depth 1 --branch develop https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
299 299 # Shallow partial clone fallback
300 -git clone --depth 1 --filter=tree:0 https://github.com/user/repo ~/.pi/research/repos/github.com.user.repo
300 +git clone --depth 1 --filter=tree:0 https://github.com/user/repo ~/.local/state/pi/research/repos/github.com.user.repo
303 303 ## Pre-flight Check
312 312 which rg # ripgrep (optional, faster)
314 314 # List existing research
315 -ls ~/.pi/research/pages/
316 -ls ~/.pi/research/repos/
315 +ls ~/.local/state/pi/research/pages/
316 +ls ~/.local/state/pi/research/repos/
319 319 ## Cleanup (Optional)
323 323 ```bash
324 324 # Remove single item
325 -rm -rf ~/.pi/research/repos/github.com.user.repo
326 -rm ~/.pi/research/pages/example.com.html
325 +rm -rf ~/.local/state/pi/research/repos/github.com.user.repo
326 +rm ~/.local/state/pi/research/pages/example.com.html
328 328 # Remove by age (e.g., older than 30 days)
329 -find ~/.pi/research -type f -mtime +30 -delete
329 +find ~/.local/state/pi/research -type f -mtime +30 -delete
331 331 # Remove empty dirs
332 -find ~/.pi/research -type d -empty -delete
332 +find ~/.local/state/pi/research -type d -empty -delete
335 335 **Default:
342 342 **Fix:** Use separate mkdir commands, not brace expansion:
344 344 ```bash
345 -mkdir -p ~/.pi/research/pages
346 -mkdir -p ~/.pi/research/repos
347 -mkdir -p ~/.pi/research/extracts
345 +mkdir -p ~/.local/state/pi/research/pages
346 +mkdir -p ~/.local/state/pi/research/repos
347 +mkdir -p ~/.local/state/pi/research/extracts
350 350 ### Issue: "monolith fails"
352 352 **Fix:** Fallback to curl:
354 354 ```bash
355 -curl -sL "https://example.com" -o ~/.pi/research/pages/example.com.html
355 +curl -sL "https://example.com" -o ~/.local/state/pi/research/pages/example.com.html
358 358 ### Issue: "grep finds nothing in HTML"
361 361 Try ripgrep with `-s` (no warnings):
363 363 ```bash
364 -rg -s "pattern" ~/.pi/research/pages/
364 +rg -s "pattern" ~/.local/state/pi/research/pages/
365 365 # Or search for text within specific tags
366 -rg ">([^<]*pattern[^<]*)<" ~/.pi/research/pages/
366 +rg ">([^<]*pattern[^<]*)<" ~/.local/state/pi/research/pages/
369 369 ### Issue: "git clone hangs"
371 371 **Fix:** Use timeout and retry:
373 373 ```bash
374 -timeout 60 git clone --depth 1 URL ~/.pi/research/repos/name || retry
374 +timeout 60 git clone --depth 1 URL ~/.local/state/pi/research/repos/name || retry
377 377 ## Pro Tips
381 381 - **Use monolith for complexity**:
382 382 SPAs, pages needing JS rendering, archival
383 383 - **Check before clone**:
384 - `ls ~/.pi/research/repos/` might already have it
384 + `ls ~/.local/state/pi/research/repos/` might already have it
385 385 - **Shallow first**:
386 386 Always `--depth 1` unless history needed
387 387 - **Sparse when sure**:
398 398 ```bash
399 399 # Setup
400 -mkdir -p ~/.pi/research/pages
401 -mkdir -p ~/.pi/research/repos
402 -mkdir -p ~/.pi/research/extracts
400 +mkdir -p ~/.local/state/pi/research/pages
401 +mkdir -p ~/.local/state/pi/research/repos
402 +mkdir -p ~/.local/state/pi/research/extracts
404 404 # Check state
405 -ls ~/.pi/research/pages/
406 -ls ~/.pi/research/repos/
405 +ls ~/.local/state/pi/research/pages/
406 +ls ~/.local/state/pi/research/repos/
408 408 # Web (monolith - primary)
409 -monolith URL -o ~/.pi/research/pages/out.html
410 -monolith -a -c -F -i -j -M -I URL -o ~/.pi/research/pages/out.html # Clean extract
409 +monolith URL -o ~/.local/state/pi/research/pages/out.html
410 +monolith -a -c -F -i -j -M -I URL -o ~/.local/state/pi/research/pages/out.html # Clean extract
412 412 # Web (curl - fallback)
413 -curl -sL URL -o ~/.pi/research/pages/out.html
413 +curl -sL URL -o ~/.local/state/pi/research/pages/out.html
416 -git clone --depth 1 URL ~/.pi/research/repos/name # Shallow
416 +git clone --depth 1 URL ~/.local/state/pi/research/repos/name # Shallow
417 417 git sparse-checkout set path path2 # Selective
418 -git clone --depth 1 --filter=tree:0 --sparse URL ~/.pi/research/repos/name # Both
418 +git clone --depth 1 --filter=tree:0 --sparse URL ~/.local/state/pi/research/repos/name # Both
420 420 # Search
421 -rg "pattern" ~/.pi/research/pages/
422 -rg "pattern" ~/.pi/research/repos/
423 -rg "pattern" ~/.pi/research/ # Everything
421 +rg "pattern" ~/.local/state/pi/research/pages/
422 +rg "pattern" ~/.local/state/pi/research/repos/
423 +rg "pattern" ~/.local/state/pi/research/ # Everything