repositories / dotfiles

dotfiles

bugabingas dorkfiles

owned by bugabinga

pi: migrate fxdriver to managed skill family

7bd93a84eb26ad8302c64bec124b31218b04f24f

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (22)

pi/agent/settings.json

Modified
8 8 "skills": [
9 - "-skills/fxdriver-instructions/SKILL.md",
10 - "-skills/fxdriver/SKILL.md",
9 + "-skills/fxdriver/fxdriver-instructions/SKILL.md",
10 + "-skills/fxdriver/fxdriver/SKILL.md",
11 11 "-skills/java/java-class-file-api/SKILL.md",
12 12 "-skills/java/java-ffm/SKILL.md",
13 13 "-skills/java/java-final-field-mutation/SKILL.md",

pi/agent/skills/.isp-skills.json

Modified
2 - "experimentation": "4377b7dd6180967d47b1011aafd7d117129d46562388dce5778d0467679da200",
3 - "fxdriver": "e0886ac9c2b149233b28925e92e711967c850194a0b4137521f0c35371bbce49",
4 - "fxdriver-instructions": "03acd906bfa49e7feef159191988e105505c9b5a6a1f80d7e48f1b2cf587665f",
5 - "isp-start-day": "36e3ed2f300e58b365b7a25bd21d8914f066d17735b188c7cd479172ecda82f4",
2 + "experimentation": "4845509810da12c21848c88dbeefae4f9258f0d49a45af88bca4b7ab3b481465",
3 + "fxdriver": "0f1c9787da6f6904c676c26fa2fc8157159f9b7dac65a27c092e7b30c876064a",
6 4 "desloppy": "0df4ab44f58e464958d3d576533d2d5cf66359986c8175d45c5c4b2541fe13bc",
7 5 "security": "0d34727da07bb012f1f741c73c2f0cd77456435443107de7da5e231b98fd224e",
8 - "uuid-generator": "c713df36554aceff0c2a3b094502c1520a3a3df48198418721cb009840fda625"
6 + "writing-skills": "7e9a232978274760f71b29116d9c289dc2320267f94c461e25105ed56e4db82c"

pi/agent/skills/fxdriver-instructions/SKILL.md

Deleted
deleted file mode 100644
1 ----
2 -name: fxdriver-instructions
3 -description:
4 - "Clarify and harden human instructions before using fxdriver to run a real
5 - JavaFX app workflow, especially when credentials, data setup, app state, or
6 - video evidence are involved."
7 -license: MIT
8 -compatibility:
9 - "Use alongside the fxdriver skill; this skill is about test instructions and
10 - acceptance clarity, not low-level driver commands."
11 ----
13 -# fxdriver-instructions
15 -Use this skill before a non-trivial fxdriver run when the user's request
16 -describes an app workflow, acceptance test, demo recording, data-creation task,
17 -or real-app verification and important details are missing or ambiguous.
19 -The goal is not to slow the run down. Ask only for details that affect
20 -correctness, reproducibility, data safety, or the final evidence artifact. If a
21 -reasonable assumption is safe, proceed and state it.
23 -## Clarify Only What Matters
25 -Before driving the app, check whether the request clearly defines:
27 -- App entrypoint: binary, launcher, working directory, required environment
28 - variables.
29 -- Starting state: fresh profile/data, existing user state, reset/demo-data
30 - setup, required license or debug mode.
31 -- Credentials: username, password, first-login password reset behavior, whether
32 - remembered login is acceptable.
33 -- Data setup: manual creation vs fixture/demo generator, exact counts vs "at
34 - least", total counts vs per-parent counts, names/identifiers that should be
35 - searchable later.
36 -- Workflow evidence: which screens must be shown, whether searches must merely
37 - show results or open found entities, which dialogs must be visible at the end.
38 -- Artifacts: output directory, GIF/MP4/both, target window size/canvas, whether
39 - step titles are desired, naming convention for files.
40 -- Cleanup: leave the app running, close gracefully, kill launched process,
41 - preserve or discard generated data.
42 -- Workarounds: whether attach agents, debug hooks, internal reset buttons, or
43 - app restarts are allowed when fxdriver primitives cannot drive the requested
44 - UI directly.
46 -## Ask Compactly
48 -Prefer one short grouped question instead of a long interview. Good examples:
50 -```text
51 -Before I run this: should I use a fresh profile, is generated fixture data
52 -acceptable for the requested entities, and where should I write the final
53 -video?
54 -```
56 -```text
57 -Two details affect correctness: do you need exactly 5 entities total or 5 per
58 -parent, and should the search step open the found records or only show result
59 -lists?
60 -```
62 -If the user already gave enough detail, do not ask. Execute.
64 -## Make Safe Assumptions Explicit
66 -When proceeding without clarification, state the assumption in one sentence:
68 -```text
69 -I will write artifacts under the app repo's target directory and close only the
70 -process I launch.
71 -```
73 -```text
74 -I will use generated fixture data because the request asks for realistic data,
75 -not hand-entered field-by-field data.
76 -```
78 -## Escalate Ambiguity
80 -Ask before acting when a wrong assumption could invalidate the run:
82 -- Destructive data reset or production-like data.
83 -- Ambiguous credentials or password reset.
84 -- Exact entity counts when the app UI exposes per-parent counts.
85 -- Artifact location that the user will immediately consume.
86 -- Internal/debug workarounds that alter normal user behavior.
88 -## Pair With fxdriver
90 -After instructions are clear enough, switch to the `fxdriver` skill for the
91 -actual observe/act/wait loop. For long video evidence, prefer fixed-canvas
92 -`videoStart` with step titles:
94 -```sh
95 -java -jar fxdriver.jar rpc <port> videoStart \
96 - '{"path":"/abs/run/flow.gif","canvas":"fixed","width":1280,"height":900,"titleMode":"band","title":"Login"}'
97 -java -jar fxdriver.jar rpc <port> videoStep '{"title":"Search"}'
98 -```

pi/agent/skills/fxdriver/SKILL.md

Deleted
deleted file mode 100644
1 ----
2 -name: fxdriver
3 -description: "Drive and inspect JavaFX apps through fxdriver JSON-RPC. Use for JavaFX UI automation, assertions, screenshots, and debugging."
4 -license: MIT
5 -compatibility: "fxdriver 0.1.0-SNAPSHOT. Requires JDK 21+."
6 ----
8 -# fxdriver
10 -Drive JavaFX UI only; native dialogs and OS/global input need another tool.
12 -## Loop
14 -1. Start with `launch` or attach to a PID.
15 -2. Call `capabilities`, then `snapshot --summary`; use full `snapshot` only when needed.
16 -3. Prefer `nodeId`, then `textExact`, `eid`, role, or text. `selectorPath` is diagnostic, never a selector.
17 -4. Act with the control-specific method: `fire`/`click`, `setText`/`type`, `setValue`, `selectIndex`, `fireMenuItem`, `tableCell`, or `key`.
18 -5. `wait`/`assert` the result. Capture and read screenshots for visual claims.
19 -6. Save `events` on failure; call `shutdown`—launched apps exit, attached apps stay running.
21 -## Launch lifecycle
23 -`launch` supervises the app and blocks until it exits. Run it as a long-lived background process, redirect stdout/stderr, and wait for its first JSON line. Do not timeout or kill the launcher while driving the app: its shutdown hook kills the app.
25 -```sh
26 -java -jar fxdriver.jar launch --quiet -- java [options] app.Main > target/fxdriver-launch.json 2> target/fxdriver-launch.err &
27 -```
29 -Use that JSON file directly for RPC: `java -jar fxdriver.jar rpc target/fxdriver-launch.json capabilities '{}'`; no port/token parsing.
31 -## Essential rules
33 -- Input payloads use `value`; `text` selects a node.
34 -- `key` emits synthetic JavaFX events, not OS keystrokes.
35 -- `capabilities.workingDirectory` is the app cwd; screenshot paths resolve there.
36 -- `ok:true` means the primitive ran, not that intended state changed—verify afterward.
37 -- Add semantic JavaFX ids only with permission.
38 -- Parse JSON with `jq` or Node, not Python.
39 -- Trust observable UI; do not decompile app/JAR bytecode unless driver evidence cannot answer.
40 -- Video and modal-safe asynchronous dispatch are unavailable.
42 -Read all of `references/protocol.md` in one call only when exact JSON is needed; read `references/troubleshooting.md` only after a failure.

pi/agent/skills/fxdriver/examples/basic-flow.md

Deleted
deleted file mode 100644
1 -# Basic flow
3 -Replace `<pid>` with the target JavaFX JVM PID. Use the printed port/token from
4 -`attach`.
6 -```sh
7 -./mvnw -q package
8 -java -jar fxdriver.jar attach <pid>
9 -export FXDRIVER_TOKEN=<printed-token>
10 -java -jar fxdriver.jar rpc <printed-port> ping '{}'
11 -java -jar fxdriver.jar rpc <printed-port> snapshot '{}' > target/basic-flow-snapshot.json
12 -java -jar fxdriver.jar rpc <printed-port> highlight '{"selector":"type=Button"}'
13 -java -jar fxdriver.jar screenshot \
14 - <printed-port> target/basic-flow.png > target/basic-flow-summary.json
15 -java -jar fxdriver.jar rpc <printed-port> events '{}' > target/basic-flow-events.json
16 -```

pi/agent/skills/fxdriver/examples/duplicate-review-flow.md

Deleted
deleted file mode 100644
1 -# Duplicate review flow
3 -Example sequence from the original duplicate-review app spike. Treat it as a
4 -recipe, not a script.
6 -```sh
7 -./mvnw -q package
8 -java -jar fxdriver.jar attach <pid>
9 -export FXDRIVER_TOKEN=<printed-token>
10 -java -jar fxdriver.jar rpc <printed-port> ping '{}'
11 -java -jar fxdriver.jar screenshot \
12 - <printed-port> target/dup-before.png > target/dup-before-summary.json
13 -java -jar fxdriver.jar rpc <printed-port> click '{"nodeId":"find-duplicates"}'
14 -java -jar fxdriver.jar rpc <printed-port> wait '{"text":"portrait","timeoutMs":30000}'
15 -java -jar fxdriver.jar rpc <printed-port> setText '{"nodeId":"filter-duplicates","value":"portrait"}'
16 -java -jar fxdriver.jar rpc <printed-port> selectListItem '{"nodeId":"duplicates-list","itemText":"portrait"}'
17 -java -jar fxdriver.jar rpc <printed-port> assert '{"textExact":"Keep duplicates","present":true}'
18 -java -jar fxdriver.jar screenshot \
19 - <printed-port> target/dup-detail.png > target/dup-detail-summary.json
20 -java -jar fxdriver.jar rpc <printed-port> click '{"textExact":"Keep duplicates"}'
21 -java -jar fxdriver.jar rpc <printed-port> assert '{"text":"Kept duplicates","present":true}'
22 -java -jar fxdriver.jar screenshot \
23 - <printed-port> target/dup-after.png > target/dup-after-summary.json
24 -java -jar fxdriver.jar image-diff \
25 - target/dup-before.png target/dup-after.png target/dup-diff.png \
26 - > target/dup-diff.json
27 -java -jar fxdriver.jar rpc <printed-port> events '{}' > target/dup-events.json
28 -```

pi/agent/skills/fxdriver/fxdriver-instructions/SKILL.md

Added
new file mode 100644
1 +---
2 +name: fxdriver-instructions
3 +description:
4 + "Clarify and harden human instructions before using fxdriver to run a real
5 + JavaFX app workflow, especially when credentials, data setup, app state, or
6 + video evidence are involved."
7 +license: MIT
8 +compatibility:
9 + "Use alongside the fxdriver skill; this skill is about test instructions and
10 + acceptance clarity, not low-level driver commands."
11 +---
13 +# fxdriver-instructions
15 +Use this skill before a non-trivial fxdriver run when the user's request
16 +describes an app workflow, acceptance test, demo recording, data-creation task,
17 +or real-app verification and important details are missing or ambiguous.
19 +The goal is not to slow the run down. Ask only for details that affect
20 +correctness, reproducibility, data safety, or the final evidence artifact. If a
21 +reasonable assumption is safe, proceed and state it.
23 +## Clarify Only What Matters
25 +Before driving the app, check whether the request clearly defines:
27 +- App entrypoint: binary, launcher, working directory, required environment
28 + variables.
29 +- Starting state: fresh profile/data, existing user state, reset/demo-data
30 + setup, required license or debug mode.
31 +- Credentials: username, password, first-login password reset behavior, whether
32 + remembered login is acceptable.
33 +- Data setup: manual creation vs fixture/demo generator, exact counts vs "at
34 + least", total counts vs per-parent counts, names/identifiers that should be
35 + searchable later.
36 +- Workflow evidence: which screens must be shown, whether searches must merely
37 + show results or open found entities, which dialogs must be visible at the end.
38 +- Artifacts: output directory, GIF/MP4/both, target window size/canvas, whether
39 + step titles are desired, naming convention for files.
40 +- Cleanup: leave the app running, close gracefully, kill launched process,
41 + preserve or discard generated data.
42 +- Workarounds: whether attach agents, debug hooks, internal reset buttons, or
43 + app restarts are allowed when fxdriver primitives cannot drive the requested
44 + UI directly.
46 +## Ask Compactly
48 +Prefer one short grouped question instead of a long interview. Good examples:
50 +```text
51 +Before I run this: should I use a fresh profile, is generated fixture data
52 +acceptable for the requested entities, and where should I write the final
53 +video?
54 +```
56 +```text
57 +Two details affect correctness: do you need exactly 5 entities total or 5 per
58 +parent, and should the search step open the found records or only show result
59 +lists?
60 +```
62 +If the user already gave enough detail, do not ask. Execute.
64 +## Make Safe Assumptions Explicit
66 +When proceeding without clarification, state the assumption in one sentence:
68 +```text
69 +I will write artifacts under the app repo's target directory and close only the
70 +process I launch.
71 +```
73 +```text
74 +I will use generated fixture data because the request asks for realistic data,
75 +not hand-entered field-by-field data.
76 +```
78 +## Escalate Ambiguity
80 +Ask before acting when a wrong assumption could invalidate the run:
82 +- Destructive data reset or production-like data.
83 +- Ambiguous credentials or password reset.
84 +- Exact entity counts when the app UI exposes per-parent counts.
85 +- Artifact location that the user will immediately consume.
86 +- Internal/debug workarounds that alter normal user behavior.
88 +## Pair With fxdriver
90 +After instructions are clear enough, switch to the `fxdriver` skill for the
91 +actual observe/act/wait loop. For long video evidence, prefer fixed-canvas
92 +`videoStart` with step titles:
94 +```sh
95 +java -jar fxdriver.jar rpc <port> videoStart \
96 + '{"path":"/abs/run/flow.gif","canvas":"fixed","width":1280,"height":900,"titleMode":"band","title":"Login"}'
97 +java -jar fxdriver.jar rpc <port> videoStep '{"title":"Search"}'
98 +```

pi/agent/skills/fxdriver/fxdriver.jar

Deleted
deleted file mode 100644

Previous version contains binary or non-UTF-8 data.

pi/agent/skills/fxdriver/fxdriver/SKILL.md

Added
new file mode 100644
1 +---
2 +name: fxdriver
3 +description: "Drive and inspect JavaFX apps through fxdriver JSON-RPC. Use for JavaFX UI automation, assertions, screenshots, and debugging."
4 +license: MIT
5 +compatibility: "fxdriver 0.1.0-SNAPSHOT. Requires JDK 21+."
6 +---
8 +# fxdriver
10 +Drive JavaFX UI only; native dialogs and OS/global input need another tool.
12 +## Loop
14 +1. Start with `launch` or attach to a PID.
15 +2. Call `capabilities`, then `snapshot --summary`; use full `snapshot` only when needed.
16 +3. Prefer `nodeId`, then `textExact`, `eid`, role, or text. `selectorPath` is diagnostic, never a selector.
17 +4. Act with the control-specific method: `fire`/`click`, `setText`/`type`, `setValue`, `selectIndex`, `fireMenuItem`, `tableCell`, or `key`.
18 +5. `wait`/`assert` the result. Capture and read screenshots for visual claims.
19 +6. Save `events` on failure; call `shutdown`—launched apps exit, attached apps stay running.
21 +## Launch lifecycle
23 +`launch` supervises the app and blocks until it exits. Run it as a long-lived background process, redirect stdout/stderr, and wait for its first JSON line. Do not timeout or kill the launcher while driving the app: its shutdown hook kills the app.
25 +```sh
26 +java -jar fxdriver.jar launch --quiet -- java [options] app.Main > target/fxdriver-launch.json 2> target/fxdriver-launch.err &
27 +```
29 +Use that JSON file directly for RPC: `java -jar fxdriver.jar rpc target/fxdriver-launch.json capabilities '{}'`; no port/token parsing.
31 +## Essential rules
33 +- Input payloads use `value`; `text` selects a node.
34 +- `key` emits synthetic JavaFX events, not OS keystrokes.
35 +- `capabilities.workingDirectory` is the app cwd; screenshot paths resolve there.
36 +- `ok:true` means the primitive ran, not that intended state changed—verify afterward.
37 +- Add semantic JavaFX ids only with permission.
38 +- Parse JSON with `jq` or Node, not Python.
39 +- Trust observable UI; do not decompile app/JAR bytecode unless driver evidence cannot answer.
40 +- Video and modal-safe asynchronous dispatch are unavailable.
42 +Read all of `references/protocol.md` in one call only when exact JSON is needed; read `references/troubleshooting.md` only after a failure.

pi/agent/skills/fxdriver/fxdriver/examples/basic-flow.md

Added
new file mode 100644
1 +# Basic flow
3 +Replace `<pid>` with the target JavaFX JVM PID. Use the printed port/token from
4 +`attach`.
6 +```sh
7 +./mvnw -q package
8 +java -jar fxdriver.jar attach <pid>
9 +export FXDRIVER_TOKEN=<printed-token>
10 +java -jar fxdriver.jar rpc <printed-port> ping '{}'
11 +java -jar fxdriver.jar rpc <printed-port> snapshot '{}' > target/basic-flow-snapshot.json
12 +java -jar fxdriver.jar rpc <printed-port> highlight '{"selector":"type=Button"}'
13 +java -jar fxdriver.jar screenshot \
14 + <printed-port> target/basic-flow.png > target/basic-flow-summary.json
15 +java -jar fxdriver.jar rpc <printed-port> events '{}' > target/basic-flow-events.json
16 +```

pi/agent/skills/fxdriver/fxdriver/examples/duplicate-review-flow.md

Added
new file mode 100644
1 +# Duplicate review flow
3 +Example sequence from the original duplicate-review app spike. Treat it as a
4 +recipe, not a script.
6 +```sh
7 +./mvnw -q package
8 +java -jar fxdriver.jar attach <pid>
9 +export FXDRIVER_TOKEN=<printed-token>
10 +java -jar fxdriver.jar rpc <printed-port> ping '{}'
11 +java -jar fxdriver.jar screenshot \
12 + <printed-port> target/dup-before.png > target/dup-before-summary.json
13 +java -jar fxdriver.jar rpc <printed-port> click '{"nodeId":"find-duplicates"}'
14 +java -jar fxdriver.jar rpc <printed-port> wait '{"text":"portrait","timeoutMs":30000}'
15 +java -jar fxdriver.jar rpc <printed-port> setText '{"nodeId":"filter-duplicates","value":"portrait"}'
16 +java -jar fxdriver.jar rpc <printed-port> selectListItem '{"nodeId":"duplicates-list","itemText":"portrait"}'
17 +java -jar fxdriver.jar rpc <printed-port> assert '{"textExact":"Keep duplicates","present":true}'
18 +java -jar fxdriver.jar screenshot \
19 + <printed-port> target/dup-detail.png > target/dup-detail-summary.json
20 +java -jar fxdriver.jar rpc <printed-port> click '{"textExact":"Keep duplicates"}'
21 +java -jar fxdriver.jar rpc <printed-port> assert '{"text":"Kept duplicates","present":true}'
22 +java -jar fxdriver.jar screenshot \
23 + <printed-port> target/dup-after.png > target/dup-after-summary.json
24 +java -jar fxdriver.jar image-diff \
25 + target/dup-before.png target/dup-after.png target/dup-diff.png \
26 + > target/dup-diff.json
27 +java -jar fxdriver.jar rpc <printed-port> events '{}' > target/dup-events.json
28 +```

pi/agent/skills/fxdriver/fxdriver/fxdriver.jar

Added
new file mode 100644

Current version contains binary or non-UTF-8 data.

pi/agent/skills/fxdriver/fxdriver/references/failures.md

Added
new file mode 100644
1 +# fxdriver failure examples
3 +These examples are based on integration tests that intentionally drive the
4 +probe app into failed or ambiguous states. Keep the pattern: observe the exact
5 +tool result, gather the next artifact, then change one thing.
7 +## Wait timed out
9 +Observed result:
11 +```json
13 + "result": {
14 + "query": "text=Probe buton",
15 + "present": true,
16 + "count": 0,
17 + "ok": false,
18 + "error": { "code": "TIMEOUT", "message": "Wait predicate not satisfied" }
19 + }
21 +```
23 +Useful recovery:
25 +1. Do not retry the same typo.
26 +2. Run `snapshot` or CLI `snapshot --summary`.
27 +3. Compare visible text and control ids, then retry with the corrected selector.
29 +Verified recovery signal: a full snapshot after the timeout contained
30 +`Probe button`.
32 +## Input payload was wrong
34 +Observed command:
36 +```sh
37 +java -jar fxdriver.jar rpc <port> setText '{"nodeId":"probe-field","text":"wrong field"}'
38 +```
40 +Observed result included:
42 +```text
43 +setText/type require value
44 +```
46 +Useful recovery: for `setText` and `type`, put the input in `value`. `text` is a
47 +selector field.
49 +```sh
50 +java -jar fxdriver.jar rpc <port> setText '{"nodeId":"probe-field","value":"right field"}'
51 +```
53 +## Wrong primitive for a choice control
55 +Observed behavior:
57 +1. `click {"nodeId":"probe-choice"}` returned `ok:true`.
58 +2. `wait {"text":"blue"}` still timed out.
59 +3. `setValue {"nodeId":"probe-choice","value":"blue"}` returned `ok:true`.
60 +4. `wait {"text":"blue"}` then passed.
62 +Useful recovery: `ok:true` on an action means the primitive ran, not that the
63 +intended state changed. For choice-like controls, prefer `setValue` or
64 +`selectIndex`, then wait/assert the resulting value.
66 +## Useful control has no id
68 +Observed snapshot fragment:
70 +```json
72 + "type": "Button",
73 + "id": "",
74 + "nodeId": "",
75 + "text": "Tool action",
76 + "actionable": true
78 +```
80 +Useful recovery:
82 +1. If this is a one-off run, target by exact text or role plus text and verify
83 + with `highlight`.
84 +2. If this flow will be repeated, ask permission to add a minimal semantic
85 + JavaFX id such as `tool-action`.
86 +3. Add only ids that describe user intent, follow local style, and leave them in
87 + the app for future navigation.
89 +## Visual change needs image evidence
91 +Verified sequence:
93 +```sh
94 +java -jar fxdriver.jar rpc <port> screenshot '{"path":"/abs/before.png"}'
95 +java -jar fxdriver.jar rpc <port> click '{"textExact":"charlie"}'
96 +java -jar fxdriver.jar rpc <port> wait '{"text":"clicked charlie","timeoutMs":2000}'
97 +java -jar fxdriver.jar rpc <port> screenshot '{"path":"/abs/after.png"}'
98 +java -jar fxdriver.jar image-diff /abs/before.png /abs/after.png /abs/diff.png
99 +```
101 +The screenshot RPC returned image metadata with `source`, and the files existed.
102 +`image-diff` returned `changedPixels` and wrote the diff PNG.
104 +Useful recovery: read the screenshots. The diff proves where pixels changed,
105 +not whether the UI is correct.
107 +## Use events as a trail
109 +Observed events included method names, params, and `ok` values:
111 +```json
112 +{"method":"wait","ok":false,"params":{"text":"Probe buton","timeoutMs":250}}
113 +{"method":"setText","ok":false,"params":{"nodeId":"probe-field","text":"wrong field"}}
114 +```
116 +Useful recovery: use `events {"clear":true}` at failure boundaries to preserve a
117 +short trail of what the agent actually tried. Treat `ok:false` entries as the
118 +first questions to explain.

pi/agent/skills/fxdriver/fxdriver/references/protocol.md

Added
new file mode 100644
1 +# fxdriver protocol
3 +Transport: JSON-RPC 2.0 over localhost HTTP POST.
5 +Prototype CLI:
7 +```sh
8 +java -jar fxdriver.jar attach <pid> [port]
9 +java -jar fxdriver.jar launch [port] -- \
10 + java [java-options...] <main-or-jar> [args...]
11 +java -jar fxdriver.jar rpc <port|launch-json> <method> '<params-json>' [token]
12 +java -jar fxdriver.jar snapshot <port> [--summary] [token]
13 +```
15 +Attach/launch default to port `0`, so the OS chooses a free loopback port. The
16 +first fxdriver-owned CLI output line is machine-readable JSON with `pid`,
17 +`port`, `token`, `endpoint`, and `endpointFile`. Compatibility lines still print
18 +the chosen port, an endpoint file, and `FXDRIVER_TOKEN`. RPC calls require that
19 +token via `FXDRIVER_TOKEN` or the optional final token argument. `rpc` also accepts a saved launch JSON file and reads its port/token directly.
21 +## Attach vs launch
23 +Use `attach` for already-running apps:
25 +```sh
26 +java -jar fxdriver.jar attach <pid>
27 +export FXDRIVER_TOKEN=<printed-token>
28 +```
30 +Use `launch` for repeatable runs where you control the Java command:
32 +```sh
33 +java -jar fxdriver.jar launch -- \
34 + java --module-path <mods> --add-modules javafx.controls \
35 + -cp <cp> app.Main
36 +export FXDRIVER_TOKEN=<printed-token>
37 +```
39 +Current launch mode injects `-javaagent` after `java` and supports direct Java
40 +commands only. It supervises the child and blocks until the app exits. Run it
41 +as a persistent/background process, capture its first JSON stdout line, and keep
42 +the launcher alive: killing it also kills the app.
44 +Example launch JSON:
46 +```json
48 + "pid": 12345,
49 + "port": 43123,
50 + "token": "...",
51 + "endpoint": "http://127.0.0.1:43123/rpc",
52 + "endpointFile": "/abs/target/fxdriver-endpoints/launch-1.json",
53 + "startupMs": 822
55 +```
57 +## Core methods
59 +### `ping`
61 +```sh
62 +java -jar fxdriver.jar rpc <port> ping '{}'
63 +```
65 +Returns liveness.
67 +### `version`, `capabilities`
69 +```sh
70 +java -jar fxdriver.jar rpc <port> version '{}'
71 +java -jar fxdriver.jar rpc <port> capabilities '{}'
72 +```
74 +Use these before relying on optional methods or selector forms. `capabilities` also returns `workingDirectory`, the base for relative screenshot paths.
76 +### `configure`
78 +```sh
79 +java -jar fxdriver.jar rpc <port> configure '{"visualTrace":true,"highlightMs":250,"effect":"invert"}'
80 +java -jar fxdriver.jar rpc <port> configure '{"visualTrace":false}'
81 +```
83 +Controls optional visual tracing. When enabled, supported actions highlight the
84 +resolved target briefly before acting. Default is off. Effects are
85 +non-layout-affecting and apply to any JavaFX `Node`: `invert` (default;
86 +theme-independent), `colorAdjust`, `glow`, `bloom`.
88 +Per-action override:
90 +```sh
91 +java -jar fxdriver.jar rpc <port> click '{"nodeId":"save","highlightMs":250}'
92 +```
94 +`highlightMs:0` disables trace for that action.
96 +### `run`, batch, and `mark`
98 +Group dependent actions with `run`; each step uses `op` (or `method`) plus the
99 +normal method parameters. It stops on the first failed step unless
100 +`continueOnError:true`. `returnState:"compact"` adds bounded state.
102 +```sh
103 +java -jar fxdriver.jar rpc <port> run '{"steps":[{"op":"setText","nodeId":"name","value":"Ada"},{"op":"wait","textExact":"Ada"}],"returnState":"compact"}'
104 +java -jar fxdriver.jar rpc <port> mark '{}'
105 +```
107 +The HTTP endpoint also accepts a raw JSON-RPC request array as a batch. `mark`
108 +visually marks showing stages and needs no parameters.
110 +### `snapshot`
112 +```sh
113 +java -jar fxdriver.jar rpc <port> snapshot '{}'
114 +java -jar fxdriver.jar snapshot <port> --summary
115 +```
117 +Optional type filters (exact simple class name, case-sensitive):
119 +```sh
120 +java -jar fxdriver.jar rpc <port> snapshot '{"includeTypes":["Button","TextField"]}'
121 +java -jar fxdriver.jar rpc <port> snapshot '{"excludeTypes":["LabeledText","FontIcon"]}'
122 +```
124 +Both can be combined. `excludeTypes` wins over `includeTypes`. Default (no
125 +filters) returns all visible nodes.
127 +Returns:
129 +- `windows[]`: visible JavaFX windows with bounds and nested nodes.
130 +- `nodes[]`: flat compatibility list.
132 +Important node fields:
134 +- `eid` / `handle`
135 +- `parent`
136 +- `window`
137 +- `type`
138 +- `id`
139 +- `text`
140 +- `accessibleText`, `accessibleHelp`, `accessibleRole`,
141 + `accessibleRoleDescription`
142 +- `styleClasses`
143 +- `visible`, `disabled`, `focused`, `managed`, `mouseTransparent`,
144 + `pickOnBounds`, `opacity`, `cursor`, `actionable`
145 +- `bounds` (screen coordinates), `boundsInLocal`, `layoutBounds`
146 +- optional `semantic` object
148 +Initial `semantic` support:
150 +- `kind`: `labeled`, `button`, `hyperlink`, `toggle`, `checkbox`, `radio`,
151 + `textInput`, `list`, `tree`, `table`, `treeTable`, `comboBox`, `choiceBox`,
152 + `spinner`, `tabPane`, `text`, `control`
153 +- `ComboBox`/`ChoiceBox`: `value`, `itemsCount`, `selectedIndex` (+
154 + `editable`/`showing` for ComboBox); `Spinner`: `value`, `editable`; `TabPane`:
155 + `value` (selected tab text), `tabsCount`, `selectedIndex`
156 +- `Labeled`: `text`, `graphicType`, `contentDisplay`, `mnemonicParsing`,
157 + `wrapText`, `textOverrun`
158 +- `ButtonBase`: `armed`; `Button`: `defaultButton`, `cancelButton`; `Hyperlink`:
159 + `visited`
160 +- `ToggleButton`/`RadioButton`: `selected`, `toggleGroup`
161 +- `CheckBox`: `selected`, `indeterminate`, `allowIndeterminate`
162 +- `TextInputControl`: `value`, `prompt`, `editable`, `selectedText`,
163 + `selectionStart`, `selectionEnd`, `caretPosition`, `anchor`, `length`,
164 + `masked`
165 +- `TextField`: `prefColumnCount`; `TextArea`: `prefColumnCount`, `prefRowCount`,
166 + `wrapText`, `scrollTop`, `scrollLeft`
167 +- `ListView`: `itemsCount`, `selectionMode`, `selectedIndices`, `selectedItems`,
168 + `focusedIndex`, `editable`, `fixedCellSize`, `orientation`, `placeholderText`
169 +- `TreeView`: `itemsCount` (expanded items), `selectionMode`, `selectedIndices`,
170 + `selectedItems`, `focusedIndex`, `editable`, `fixedCellSize`, `rootVisible`,
171 + `rootValue`
172 +- `TableView`: `itemsCount`, `selectionMode`, `selectedIndices`,
173 + `selectedItems`, `focusedIndex`, `focusedCell`, `editable`, `fixedCellSize`,
174 + `placeholderText`, `columns`, `sortOrder`
175 +- `TreeTableView`: `itemsCount` (expanded items), `selectionMode`,
176 + `selectedIndices`, `selectedItems`, `focusedIndex`, `focusedCell`, `editable`,
177 + `fixedCellSize`, `rootVisible`, `rootValue`, `placeholderText`, `columns`,
178 + `sortOrder`
180 +`PasswordField` text/value/selectedText are redacted by default
181 +(`masked: true`). Virtualized controls expose model/selection metadata;
182 +unrealized rows/cells are not emitted as nodes.
184 +### Snapshot summary
186 +```sh
187 +java -jar fxdriver.jar snapshot <port> --summary
188 +```
190 +The CLI summary command returns a compact first view. It caps windows at 8,
191 +buttons at 24, text fields at 16, value controls at 24, menus at 16, tables at
192 +12, selected tabs at 12, and the distinct visible-text sample at 40 strings. Integration tests enforce a 16 KiB
193 +response bound for the focused fixture.
195 +- `windows[]`: window id/title/focus/bounds.
196 +- `buttons[]`: visible `ButtonBase` controls with text and disabled state.
197 +- `textFields[]`: text inputs with value/prompt/editability.
198 +- `controls[]`: toggles and choice/date/spinner/range values.
199 +- `menus[]`: menu controls and context-menu targets with nested item paths.
200 +- `tables[]`: visible `TableView` and `TreeTableView` row/column summaries.
201 +- `selectedTabs[]`: selected tab text per visible `TabPane`.
202 +- `focusedNode`: target metadata or `null`.
203 +- `visibleTextSample[]`: bounded visible text for orientation.
205 +Use full `snapshot` when you need complete semantics or raw node hierarchy.
207 +### `highlight`
209 +```sh
210 +java -jar fxdriver.jar rpc <port> highlight '{"selector":"type=Button"}'
211 +java -jar fxdriver.jar rpc <port> highlight '{"selector":"type=Button","effect":"invert"}'
212 +java -jar fxdriver.jar rpc <port> highlight '{"selector":"type=Button","effect":"glow"}'
213 +```
215 +Applies a temporary non-layout JavaFX `Effect` to raw matching `Node`s,
216 +including non-control nodes such as `Text`, layout panes, and `SubScene`.
217 +Default `invert` effect uses blend-difference against white over the node's
218 +local bounds, so it is visible on both light and dark themes without changing
219 +layout. Use to verify selector ambiguity or show humans what target will be
220 +used. Calling `highlight` with no matches clears prior highlights. Visual trace
221 +for actions highlights the resolved actionable target instead.
223 +### `click`
225 +```sh
226 +java -jar fxdriver.jar rpc <port> click '{"nodeId":"find-duplicates"}'
227 +java -jar fxdriver.jar rpc <port> click '{"textExact":"Keep duplicates"}'
228 +java -jar fxdriver.jar rpc <port> click '{"textExact":"Projects","untilQuietMs":500,"timeoutMs":10000}'
229 +```
231 +Fires `ButtonBase` directly when possible; otherwise uses JavaFX Robot center
232 +click. With `untilQuietMs`, waits after the click until the visible UI signature
233 +has stayed stable for that many milliseconds, or returns `QUIET_TIMEOUT`.
235 +### `fire`
237 +```sh
238 +java -jar fxdriver.jar rpc <port> fire '{"textExact":"Find duplicates..."}'
239 +```
241 +Calls `ButtonBase.fire()` on first matching button-like control.
243 +### `type`, `setText`, `clear`
245 +```sh
246 +java -jar fxdriver.jar rpc <port> setText '{"nodeId":"filter-duplicates","value":"portrait"}'
247 +java -jar fxdriver.jar rpc <port> clear '{"nodeId":"filter-duplicates"}'
248 +java -jar fxdriver.jar rpc <port> type '{"nodeId":"filter-duplicates","value":"abc","replace":false}'
249 +```
251 +Prefer `setText` for deterministic tests; use `type` when simulating user input
252 +matters. These set the control's text directly; for real keystrokes (Enter to
253 +submit a dialog, Tab to move focus, accelerators), use `key`.
255 +### `key`
257 +```sh
258 +java -jar fxdriver.jar rpc <port> key '{"key":"ENTER"}'
259 +java -jar fxdriver.jar rpc <port> key '{"key":"CTRL+S"}'
260 +java -jar fxdriver.jar rpc <port> key '{"nodeId":"search","chars":"hello"}'
261 +```
263 +Dispatches synthetic JavaFX `KeyEvent`s to a node; it is not OS/global input.
264 +`key` accepts one named key or chord such as `ENTER`, `ESCAPE`/`ESC`, `TAB`,
265 +`F1`, `CTRL+S`, `SHIFT+TAB`, or `META+K`. `chars` emits one `KEY_TYPED` event
266 +per Unicode code point. With a selector (`nodeId`, `textExact`, …), the first
267 +match is focused; otherwise the current JavaFX focus owner is used. The result
268 +contains `key`, `chars`, `sent`, and `target`; no target returns `NO_FOCUS`.
270 +### `setValue`, `selectIndex`, `selectListItem`
272 +```sh
273 +java -jar fxdriver.jar rpc <port> setValue '{"nodeId":"country","value":"Germany"}'
274 +java -jar fxdriver.jar rpc <port> selectIndex '{"nodeId":"projects","index":2}'
275 +java -jar fxdriver.jar rpc <port> selectListItem '{"nodeId":"duplicates-list","itemText":"portrait"}'
276 +```
278 +Drives controls that `click`/`fire` cannot reliably set. Use `setValue` for
279 +controls with a direct value such as `ChoiceBox`, `ComboBox`, `DatePicker`,
280 +`Slider`, and text-like value controls. Use `selectIndex` for indexed selection
281 +in list/table/tree-style controls. Use `selectListItem` for `ListView` by item
282 +text. A generic `select` method is intentionally unsupported.
284 +### Choice, popup, and step controls
286 +```sh
287 +java -jar fxdriver.jar rpc <port> setValue '{"nodeId":"theme","value":"Dark"}'
288 +java -jar fxdriver.jar rpc <port> showPopup '{"nodeId":"density"}'
289 +java -jar fxdriver.jar rpc <port> hidePopup '{"nodeId":"density"}'
290 +java -jar fxdriver.jar rpc <port> increment '{"nodeId":"days","steps":5}'
291 +java -jar fxdriver.jar rpc <port> decrement '{"nodeId":"days","steps":2}'
292 +```
294 +`setValue` supports choice/combo/date/range/spinner controls. Increment and
295 +decrement default to one step.
297 +### `tableCell`
299 +```sh
300 +java -jar fxdriver.jar rpc <port> tableCell '{"nodeId":"projects","tableText":"MDP_40982497","column":"Projektnummer"}'
301 +java -jar fxdriver.jar rpc <port> tableCell '{"nodeId":"projects","rowIndex":2,"columnIndex":0}'
302 +```
304 +Reads `TableView` or `TreeTableView` model values by row text/row index and
305 +flat top-level column text, id, or index. It does not depend on realized cell
306 +nodes. The result includes `rowIndex`, `columnIndex`, resolved `column`, full
307 +`rowText`, `value`, and `ok`; failures use `NO_TABLE`, `NO_ROW`, or `NO_COLUMN`.
309 +### `fireMenuItem`
311 +```sh
312 +java -jar fxdriver.jar rpc <port> fireMenuItem '{"nodeId":"main-menu","path":["File","Save"]}'
313 +java -jar fxdriver.jar rpc <port> fireMenuItem '{"nodeId":"actions","itemText":"Delete"}'
314 +```
316 +Fires `MenuBar`, `MenuButton`, `SplitMenuButton`, or context-menu items without
317 +showing a JavaFX popup. It cannot operate native menus or dialogs. Use `path`
318 +for nested traversal or `itemText` for exact-then-contains recursive lookup.
319 +The result contains `query`, `kind`, `path`, `value`, and `fired`; failures use
320 +`NO_MENU_ITEM` or `DISABLED`.
322 +### `wait`
324 +```sh
325 +java -jar fxdriver.jar rpc <port> wait '{"text":"Done","timeoutMs":5000}'
326 +java -jar fxdriver.jar rpc <port> wait '{"nodeId":"save","enabled":true,"timeoutMs":5000}'
327 +```
329 +Polls until selector is present by default. Default timeout is 2000 ms; pass
330 +explicit `timeoutMs` for slow app operations. Optional predicates: `present`,
331 +`enabled`, `focused`, `visible`. Like the action commands, `wait` and `assert`
332 +only consider visible nodes unless `visible:false` is passed explicitly, so a
333 +satisfied `wait` means the follow-up `click`/`type` sees the same node. Failed
334 +waits return `ok:false` with `TIMEOUT` and up to five bounded `nearMatches`.
335 +Each candidate has `eid`, `type`, `id`, `text`, and diagnostic `selectorPath`.
336 +Near matches and selector paths explain misses; they are not supported
337 +selectors. Successful waits omit `nearMatches`.
339 +### `assert`
341 +```sh
342 +java -jar fxdriver.jar rpc <port> assert '{"textExact":"Keep duplicates","present":true}'
343 +java -jar fxdriver.jar rpc <port> assert '{"selector":"type=Button","count":3}'
344 +```
346 +Returns `{ok,count}`. The CLI exits non-zero only on a protocol-level JSON-RPC
347 +error (top-level `error` member); app-level `ok:false` results exit zero and
348 +need inspection in scripts.
350 +### `events`, `eventsSince`, `clearEvents`
352 +```sh
353 +java -jar fxdriver.jar rpc <port> events '{}'
354 +java -jar fxdriver.jar rpc <port> eventsSince '{"cursor":20}'
355 +java -jar fxdriver.jar rpc <port> eventsSince '{"since":1735689600000}'
356 +java -jar fxdriver.jar rpc <port> events '{"clear":true}'
357 +java -jar fxdriver.jar rpc <port> clearEvents '{}'
358 +```
360 +Returns recent fxdriver calls with `ts`, `method`, `ok`, and `params`. If
361 +request params include `tag` or `scenario`, the event copies them to top-level
362 +fields. `eventsSince` accepts an event cursor or millisecond timestamp. Use `events {"clear":true}` to read and clear in one call.
364 +### `screenshot`
366 +CLI shortcut for the focused showing stage, then first showing stage:
368 +```sh
369 +java -jar fxdriver.jar screenshot <port> target/fxdriver.png
370 +```
372 +RPC supports window/default, node, and rect targets:
374 +```sh
375 +java -jar fxdriver.jar rpc <port> screenshot '{"path":"/tmp/run/window.png"}'
376 +java -jar fxdriver.jar rpc <port> screenshot '{"target":"node","nodeId":"save","path":"/tmp/run/save.png"}'
377 +java -jar fxdriver.jar rpc <port> screenshot '{"target":"rect","x":0,"y":0,"w":320,"h":200,"path":"/tmp/run/crop.png"}'
378 +java -jar fxdriver.jar rpc <port> screenshot '{"path":"/tmp/run/dialog.png","window":"w2"}'
379 +```
381 +A relative `path` resolves against the target app's working directory, not the
382 +caller's — pass an absolute path (or use the CLI `screenshot` subcommand, which
383 +absolutizes before sending). The result echoes the absolute path written. Window
384 +and rect targets default to the focused showing JavaFX stage, then the first
385 +showing stage; pass `window` with a stage `wid` from snapshot (`"w2"`) to
386 +capture that exact stage. Popup window ids are rejected. `press` accepts the
387 +same parameter to scope accelerators.
389 +Captures optimized lossless PNG. RPC results include `path`, `bytes`, `target`,
390 +`image` bounds, `source` bounds/units, `scale`, `activeWindow`, and a bounded
391 +`visibleTextSample` for orientation. `activeWindow` contains only id, title,
392 +bounds, and focused-node metadata—not a recursive snapshot. CLI screenshot
393 +prints that result plus image summary. Opaque screenshots may be encoded as
394 +RGB; transparent screenshots preserve alpha.
396 +### Video and modal dispatch
398 +`videoStart`, `videoStep`, and `videoStop` are not available in this phase.
399 +Modal-safe asynchronous action dispatch is also deferred. Check `capabilities`
400 +rather than assuming either API exists.
402 +### `image-summary`
404 +```sh
405 +java -jar fxdriver.jar image-summary target/fxdriver.png
406 +```
408 +Reports path, bytes, width, height, average color, and average hash.
410 +### `image-diff`
412 +```sh
413 +java -jar fxdriver.jar image-diff before.png after.png target/diff.png
414 +```
416 +Reports changed pixels, percent, changed bounds, size mismatch, and optional
417 +optimized lossless diff PNG path.
419 +### `listItems`
421 +```sh
422 +java -jar fxdriver.jar rpc <port> listItems '{"nodeId":"duplicates-list"}'
423 +```
425 +Returns `ListView` item strings.
427 +### `selectListItem`
429 +```sh
430 +java -jar fxdriver.jar rpc <port> selectListItem '{"nodeId":"duplicates-list","itemText":"portrait"}'
431 +```
433 +Selects matching item. Default `activate=true` also fires a matching realized
434 +row button if available.
436 +### `scroll`
438 +```sh
439 +java -jar fxdriver.jar rpc <port> scroll '{"selector":"type=ScrollPane","amount":6}'
440 +```
442 +Uses control-specific scroll when possible; Robot fallback otherwise.
444 +### `shutdown`
446 +```sh
447 +java -jar fxdriver.jar rpc <port> shutdown '{}'
448 +```
450 +Stops the RPC server. Apps started by `launch` exit; attached apps keep running. `exitApp:true` explicitly exits either.
452 +## Selectors
454 +Supported forms:
456 +```text
457 +@handle capability label for numeric handles
458 +@42 handle
459 +eid=n42 node eid
460 +#node-id JavaFX node id
461 +.style-class JavaFX style class
462 +=exact text exact visible/control text
463 +text=Open contains visible/control text
464 +regexText=Open.* regex over visible/control text
465 +type=Button Java class simple name
466 +role=BUTTON accessible role
467 +accessible=Save accessible text contains
468 +```
470 +JSON params that map to selectors:
472 +```json
473 +{ "handle": 42 }
474 +{ "nodeId": "save" }
475 +{ "cssId": "save" }
476 +{ "styleClass": "primary" }
477 +{ "textExact": "Save" }
478 +{ "text": "Save" }
479 +{ "regexText": "Sa.*" }
480 +{ "type": "Button" }
481 +{ "role": "BUTTON" }
482 +{ "accessible": "Save" }
483 +{ "selector": "type=Button" }
484 +```
486 +Snapshot and near-match `selectorPath` values are diagnostics only and cannot
487 +be passed back as selectors. Generic `select` is unsupported; use the explicit
488 +selection methods above.

pi/agent/skills/fxdriver/fxdriver/references/troubleshooting.md

Added
new file mode 100644
1 +# troubleshooting fxdriver
3 +## Attach fails
5 +Check PID:
7 +```sh
8 +jcmd -l
9 +```
11 +Common causes:
13 +- process exited;
14 +- wrong PID;
15 +- target is not a compatible JVM;
16 +- dynamic attach restricted;
17 +- target runtime image lacks attach support;
18 +- port already in use.
20 +For the current prototype, launching the app with debug flags has worked:
22 +```sh
23 +-XX:+EnableDynamicAgentLoading
24 +--add-modules=java.instrument
25 +```
27 +Do not assume final distribution will require these; integration remains open.
29 +## RPC doesn't respond
31 +Check attach output for endpoint/port.
33 +Try:
35 +```sh
36 +java -jar target/fxdriver.jar rpc <port> ping '{}'
37 +```
39 +If port is closed, reattach or inspect app stderr.
41 +## Selector matches too much
43 +Text queries may match both a control and skin/internal text. Prefer:
45 +```json
46 +{ "nodeId": "save" }
47 +{ "textExact": "Save" }
48 +{ "selector": "role=BUTTON" }
49 +```
51 +Use highlight:
53 +```sh
54 +java -jar target/fxdriver.jar rpc <port> highlight '{"text":"Save"}'
55 +```
57 +## Click doesn't do expected thing
59 +Prefer semantic/direct actions when available:
61 +- inspect `semantic.kind`/state before acting;
62 +- `fire` for `ButtonBase`;
63 +- `setText` for `TextInputControl`;
64 +- `selectListItem` for `ListView`.
66 +Robot click can be backend/session sensitive, especially Wayland.
68 +## Text/password fields
70 +`PasswordField` snapshot text/value is redacted by default (`masked: true`). Use app-specific test hooks if tests must verify actual secrets; do not expect password values in snapshots.
72 +## Virtualized controls
74 +`ListView`, `TreeView`, `TableView`, and `TreeTableView` only realize visible cells. Snapshot shows visible nodes, not all model rows. Use `semantic` metadata for counts, selected indices/items, focus, root value, and table columns.
76 +Use model primitives where available:
78 +```sh
79 +java -jar target/fxdriver.jar rpc <port> listItems '{"nodeId":"my-list"}'
80 +java -jar target/fxdriver.jar rpc <port> selectListItem '{"nodeId":"my-list","itemText":"needle"}'
81 +```
83 +## Visual bugs
85 +JSON assertions can pass while layout is visually wrong. Capture screenshots:
87 +```sh
88 +java -jar target/fxdriver.jar screenshot <port> target/failure.png
89 +```
91 +Read the screenshot. Use `image-diff` to localize changes, not to decide quality alone.
93 +## Native dialogs
95 +Native OS file/color dialogs are outside JavaFX scene graph. They are not MVP. Prefer app test mode or avoid native dialogs in fxdriver flows for now.
97 +## WebView
99 +MVP treats `WebView` as a JavaFX node with bounds. DOM-level automation is later.

pi/agent/skills/fxdriver/fxdriver/references/visual.md

Added
new file mode 100644
1 +# visual artifacts
3 +fxdriver visual primitives are inputs for agent judgment, not a built-in quality
4 +policy.
6 +## Screenshot
8 +```sh
9 +java -jar fxdriver.jar screenshot <port> target/ui.png
10 +```
12 +Output shape:
14 +```json
16 + "screenshot": {
17 + "jsonrpc": "2.0",
18 + "result": {
19 + "path": "...",
20 + "bytes": 12345,
21 + "activeWindow": { "title": "Main" },
22 + "visibleTextSample": ["Search", "Save"]
23 + }
24 + },
25 + "summary": {
26 + "path": "...",
27 + "bytes": 12345,
28 + "width": 1256,
29 + "height": 1408,
30 + "avgColor": { "r": 241, "g": 239, "b": 232, "a": 255 },
31 + "ahash": "feffdfdf7f7f7f3b"
32 + }
34 +```
36 +Read the image when making visual/layout/color claims. Screenshots are optimized
37 +lossless PNGs by default (max PNG compression where available, RGB when fully
38 +opaque). To avoid context bloat, prefer the smallest screenshot that can answer
39 +the question once node/rect screenshots exist; keep full-size PNGs on disk for
40 +audit/diff truth.
42 +## Image summary
44 +```sh
45 +java -jar fxdriver.jar image-summary target/ui.png
46 +```
48 +Use for quick sanity:
50 +- dimensions non-zero;
51 +- file size plausible;
52 +- hash changed/didn't change;
53 +- average color roughly expected.
55 +Do not use average color/hash as proof of layout quality.
57 +## Image diff
59 +```sh
60 +java -jar fxdriver.jar image-diff target/before.png target/after.png target/diff.png
61 +```
63 +Output fields:
65 +- `changedPixels`
66 +- `changedPercent`
67 +- `bounds`
68 +- `sizeMismatch`
69 +- `diff` path if requested
71 +Use diff to find if something changed and where. Then inspect `after.png` and/or
72 +`diff.png`. Diffs are meaningful only when images share compatible source rect,
73 +scale, and dimensions.
75 +## Context-efficient visual workflow
77 +Use this progressive order:
79 +1. `snapshot --summary` for windows/titles/key controls.
80 +2. Full `snapshot` for structure/bounds/semantics.
81 +3. `image-summary` for dimensions/hash sanity.
82 +4. Read full screenshot only when layout/color/visual quality matters.
83 +5. Prefer cropped/node/rect screenshots when they answer the question.
84 +6. If downscaled/cropped image is unclear, request a larger crop or full PNG.
86 +Screenshot target examples:
88 +```json
89 +{"target":"node","selector":"#details","path":"target/details.png"}
90 +{"target":"rect","window":"w1","x":0,"y":0,"w":640,"h":360,"path":"target/crop.png"}
91 +```
93 +Do not use downscaled/cropped images as sole proof for tiny text, contrast, or
94 +pixel-perfect diffs. JPEG/WebP output is intentionally not planned for now; UI
95 +screenshots stay PNG.
97 +## Common visual workflow
99 +```sh
100 +java -jar fxdriver.jar screenshot <port> target/before.png
101 +# make app/code change; rebuild/relaunch/act
102 +java -jar fxdriver.jar screenshot <port> target/after.png
103 +java -jar fxdriver.jar image-diff target/before.png target/after.png target/diff.png
104 +```
106 +Checklist for human-readable UI:
108 +- text contrast readable;
109 +- icons visible on button backgrounds;
110 +- no important text clipped/truncated unexpectedly;
111 +- no controls off-screen;
112 +- toolbars wrap or scroll instead of overflowing;
113 +- panels have reasonable spacing;
114 +- selected/disabled/focus states are distinguishable;
115 +- screenshots match JSON state.
117 +fxdriver should expose primitives for this; agent/skill decides if pass/fail.

pi/agent/skills/fxdriver/fxdriver/references/workflow.md

Added
new file mode 100644
1 +# fxdriver workflow
3 +## Attach
5 +Find Java processes:
7 +```sh
8 +jcmd -l
9 +```
11 +Attach:
13 +```sh
14 +java -jar fxdriver.jar attach <pid>
15 +export FXDRIVER_TOKEN=<printed-token>
16 +java -jar fxdriver.jar rpc <printed-port> ping '{}'
17 +```
19 +## Observe
21 +Start with the compact summary when orienting:
23 +```sh
24 +java -jar fxdriver.jar snapshot <printed-port> --summary > target/fxdriver-summary.json
25 +```
27 +Use a full snapshot when the summary is not enough:
29 +```sh
30 +java -jar fxdriver.jar rpc <printed-port> snapshot '{}' > target/fxdriver-snapshot.json
31 +```
33 +Look for:
35 +- visible windows;
36 +- stable `nodeId` values;
37 +- actionable buttons/text inputs/lists;
38 +- exact text vs duplicate skin/internal text;
39 +- disabled/focused/common node state;
40 +- `semantic.kind` and control-specific state (`selected`, `value`, `prompt`,
41 + `editable`, checkbox indeterminate state, radio toggle group, list/table
42 + counts, selected indices/items, focused index, columns, etc.).
44 +## Choose selectors
46 +Preference order:
48 +1. `nodeId` / `#id`
49 +2. `eid` if stable within this attached session
50 +3. `textExact`
51 +4. `role` + text/accessibility
52 +5. `type` for broad discovery only
53 +6. bare text contains only as last resort
55 +Do not copy snapshot or near-match `selectorPath` diagnostics into requests.
56 +They are deliberately not selectors.
58 +Verify ambiguous selectors with `highlight`:
60 +```sh
61 +java -jar fxdriver.jar rpc <port> highlight '{"text":"Find"}'
62 +```
64 +If the workflow needs repeated or future automation and important controls lack
65 +stable ids, ask the human for permission to add JavaFX ids to the app under
66 +test. When approved, add the smallest useful set of semantic ids, follow local
67 +style and naming conventions, and leave them in the app. Do not add broad,
68 +mechanical ids to every node; target controls that represent user intent,
69 +navigation landmarks, form fields, menus, tables/lists, and primary actions.
71 +## Act
73 +Examples:
75 +```sh
76 +java -jar fxdriver.jar rpc <port> click '{"nodeId":"find-duplicates"}'
77 +java -jar fxdriver.jar rpc <port> click '{"textExact":"Projects","untilQuietMs":500,"timeoutMs":10000}'
78 +java -jar fxdriver.jar rpc <port> setText '{"nodeId":"filter-duplicates","value":"portrait"}'
79 +java -jar fxdriver.jar rpc <port> selectListItem '{"nodeId":"duplicates-list","itemText":"portrait"}'
80 +java -jar fxdriver.jar rpc <port> key '{"nodeId":"filter-duplicates","key":"ENTER"}'
81 +java -jar fxdriver.jar rpc <port> fireMenuItem '{"nodeId":"main-menu","path":["File","Save"]}'
82 +java -jar fxdriver.jar rpc <port> tableCell '{"nodeId":"projects","tableText":"MDP_40982497","column":"Projektnummer"}'
83 +```
85 +`key` is synthetic JavaFX input, not OS/global input. Menu firing resolves and
86 +fires the model item without showing its JavaFX popup. `tableCell` reads model
87 +values from flat columns. There is no generic `select`; choose the explicit
88 +selection method for the control.
90 +## Wait/assert
92 +After action, wait for observable state:
94 +```sh
95 +java -jar fxdriver.jar rpc <port> wait '{"text":"portrait","timeoutMs":30000}'
96 +java -jar fxdriver.jar rpc <port> assert '{"textExact":"Keep duplicates","present":true}'
97 +```
99 +Do not assume immediate UI update after action.
101 +If a wait fails, read the returned `ok:false`/`TIMEOUT` and its at-most-five
102 +`nearMatches`, then take a fresh summary or full snapshot if needed. Near-match
103 +ids/text/types help diagnose a typo; their selector paths are not selectors.
105 +## Visual review
107 +When task mentions colors, layout, overflow, visual quality, screenshots are
108 +mandatory:
110 +```sh
111 +java -jar fxdriver.jar screenshot <port> target/before.png
112 +# act/change app
113 +java -jar fxdriver.jar screenshot <port> target/after.png
114 +java -jar fxdriver.jar image-diff target/before.png target/after.png target/diff.png
115 +```
117 +Read screenshots. Diff alone tells change amount, not quality. Screenshot JSON
118 +also identifies `activeWindow` and provides a bounded `visibleTextSample` for
119 +orientation; it is not a replacement for reading the image.
121 +## Video evidence
123 +Video RPCs are not available in this phase. Check `capabilities`; use repeated
124 +screenshots when still-image evidence is sufficient. Modal-safe asynchronous
125 +action dispatch is also outside this phase.
127 +## Failure artifacts
129 +On failure, collect:
131 +```sh
132 +java -jar fxdriver.jar rpc <port> snapshot '{}' > target/failure-snapshot.json
133 +java -jar fxdriver.jar screenshot <port> target/failure.png > target/failure-summary.json
134 +java -jar fxdriver.jar rpc <port> events '{"clear":true}' > target/failure-events.json
135 +```
137 +Then inspect app logs separately.
139 +## Clean up
141 +```sh
142 +java -jar fxdriver.jar rpc <port> shutdown '{}'
143 +```
145 +This stops only fxdriver RPC server. Kill the app process separately if the run
146 +launched it.

pi/agent/skills/fxdriver/references/failures.md

Deleted
deleted file mode 100644
1 -# fxdriver failure examples
3 -These examples are based on integration tests that intentionally drive the
4 -probe app into failed or ambiguous states. Keep the pattern: observe the exact
5 -tool result, gather the next artifact, then change one thing.
7 -## Wait timed out
9 -Observed result:
11 -```json
13 - "result": {
14 - "query": "text=Probe buton",
15 - "present": true,
16 - "count": 0,
17 - "ok": false,
18 - "error": { "code": "TIMEOUT", "message": "Wait predicate not satisfied" }
19 - }
21 -```
23 -Useful recovery:
25 -1. Do not retry the same typo.
26 -2. Run `snapshot` or CLI `snapshot --summary`.
27 -3. Compare visible text and control ids, then retry with the corrected selector.
29 -Verified recovery signal: a full snapshot after the timeout contained
30 -`Probe button`.
32 -## Input payload was wrong
34 -Observed command:
36 -```sh
37 -java -jar fxdriver.jar rpc <port> setText '{"nodeId":"probe-field","text":"wrong field"}'
38 -```
40 -Observed result included:
42 -```text
43 -setText/type require value
44 -```
46 -Useful recovery: for `setText` and `type`, put the input in `value`. `text` is a
47 -selector field.
49 -```sh
50 -java -jar fxdriver.jar rpc <port> setText '{"nodeId":"probe-field","value":"right field"}'
51 -```
53 -## Wrong primitive for a choice control
55 -Observed behavior:
57 -1. `click {"nodeId":"probe-choice"}` returned `ok:true`.
58 -2. `wait {"text":"blue"}` still timed out.
59 -3. `setValue {"nodeId":"probe-choice","value":"blue"}` returned `ok:true`.
60 -4. `wait {"text":"blue"}` then passed.
62 -Useful recovery: `ok:true` on an action means the primitive ran, not that the
63 -intended state changed. For choice-like controls, prefer `setValue` or
64 -`selectIndex`, then wait/assert the resulting value.
66 -## Useful control has no id
68 -Observed snapshot fragment:
70 -```json
72 - "type": "Button",
73 - "id": "",
74 - "nodeId": "",
75 - "text": "Tool action",
76 - "actionable": true
78 -```
80 -Useful recovery:
82 -1. If this is a one-off run, target by exact text or role plus text and verify
83 - with `highlight`.
84 -2. If this flow will be repeated, ask permission to add a minimal semantic
85 - JavaFX id such as `tool-action`.
86 -3. Add only ids that describe user intent, follow local style, and leave them in
87 - the app for future navigation.
89 -## Visual change needs image evidence
91 -Verified sequence:
93 -```sh
94 -java -jar fxdriver.jar rpc <port> screenshot '{"path":"/abs/before.png"}'
95 -java -jar fxdriver.jar rpc <port> click '{"textExact":"charlie"}'
96 -java -jar fxdriver.jar rpc <port> wait '{"text":"clicked charlie","timeoutMs":2000}'
97 -java -jar fxdriver.jar rpc <port> screenshot '{"path":"/abs/after.png"}'
98 -java -jar fxdriver.jar image-diff /abs/before.png /abs/after.png /abs/diff.png
99 -```
101 -The screenshot RPC returned image metadata with `source`, and the files existed.
102 -`image-diff` returned `changedPixels` and wrote the diff PNG.
104 -Useful recovery: read the screenshots. The diff proves where pixels changed,
105 -not whether the UI is correct.
107 -## Use events as a trail
109 -Observed events included method names, params, and `ok` values:
111 -```json
112 -{"method":"wait","ok":false,"params":{"text":"Probe buton","timeoutMs":250}}
113 -{"method":"setText","ok":false,"params":{"nodeId":"probe-field","text":"wrong field"}}
114 -```
116 -Useful recovery: use `events {"clear":true}` at failure boundaries to preserve a
117 -short trail of what the agent actually tried. Treat `ok:false` entries as the
118 -first questions to explain.

pi/agent/skills/fxdriver/references/protocol.md

Deleted
deleted file mode 100644
1 -# fxdriver protocol
3 -Transport: JSON-RPC 2.0 over localhost HTTP POST.
5 -Prototype CLI:
7 -```sh
8 -java -jar fxdriver.jar attach <pid> [port]
9 -java -jar fxdriver.jar launch [port] -- \
10 - java [java-options...] <main-or-jar> [args...]
11 -java -jar fxdriver.jar rpc <port|launch-json> <method> '<params-json>' [token]
12 -java -jar fxdriver.jar snapshot <port> [--summary] [token]
13 -```
15 -Attach/launch default to port `0`, so the OS chooses a free loopback port. The
16 -first fxdriver-owned CLI output line is machine-readable JSON with `pid`,
17 -`port`, `token`, `endpoint`, and `endpointFile`. Compatibility lines still print
18 -the chosen port, an endpoint file, and `FXDRIVER_TOKEN`. RPC calls require that
19 -token via `FXDRIVER_TOKEN` or the optional final token argument. `rpc` also accepts a saved launch JSON file and reads its port/token directly.
21 -## Attach vs launch
23 -Use `attach` for already-running apps:
25 -```sh
26 -java -jar fxdriver.jar attach <pid>
27 -export FXDRIVER_TOKEN=<printed-token>
28 -```
30 -Use `launch` for repeatable runs where you control the Java command:
32 -```sh
33 -java -jar fxdriver.jar launch -- \
34 - java --module-path <mods> --add-modules javafx.controls \
35 - -cp <cp> app.Main
36 -export FXDRIVER_TOKEN=<printed-token>
37 -```
39 -Current launch mode injects `-javaagent` after `java` and supports direct Java
40 -commands only. It supervises the child and blocks until the app exits. Run it
41 -as a persistent/background process, capture its first JSON stdout line, and keep
42 -the launcher alive: killing it also kills the app.
44 -Example launch JSON:
46 -```json
48 - "pid": 12345,
49 - "port": 43123,
50 - "token": "...",
51 - "endpoint": "http://127.0.0.1:43123/rpc",
52 - "endpointFile": "/abs/target/fxdriver-endpoints/launch-1.json",
53 - "startupMs": 822
55 -```
57 -## Core methods
59 -### `ping`
61 -```sh
62 -java -jar fxdriver.jar rpc <port> ping '{}'
63 -```
65 -Returns liveness.
67 -### `version`, `capabilities`
69 -```sh
70 -java -jar fxdriver.jar rpc <port> version '{}'
71 -java -jar fxdriver.jar rpc <port> capabilities '{}'
72 -```
74 -Use these before relying on optional methods or selector forms. `capabilities` also returns `workingDirectory`, the base for relative screenshot paths.
76 -### `configure`
78 -```sh
79 -java -jar fxdriver.jar rpc <port> configure '{"visualTrace":true,"highlightMs":250,"effect":"invert"}'
80 -java -jar fxdriver.jar rpc <port> configure '{"visualTrace":false}'
81 -```
83 -Controls optional visual tracing. When enabled, supported actions highlight the
84 -resolved target briefly before acting. Default is off. Effects are
85 -non-layout-affecting and apply to any JavaFX `Node`: `invert` (default;
86 -theme-independent), `colorAdjust`, `glow`, `bloom`.
88 -Per-action override:
90 -```sh
91 -java -jar fxdriver.jar rpc <port> click '{"nodeId":"save","highlightMs":250}'
92 -```
94 -`highlightMs:0` disables trace for that action.
96 -### `run`, batch, and `mark`
98 -Group dependent actions with `run`; each step uses `op` (or `method`) plus the
99 -normal method parameters. It stops on the first failed step unless
100 -`continueOnError:true`. `returnState:"compact"` adds bounded state.
102 -```sh
103 -java -jar fxdriver.jar rpc <port> run '{"steps":[{"op":"setText","nodeId":"name","value":"Ada"},{"op":"wait","textExact":"Ada"}],"returnState":"compact"}'
104 -java -jar fxdriver.jar rpc <port> mark '{}'
105 -```
107 -The HTTP endpoint also accepts a raw JSON-RPC request array as a batch. `mark`
108 -visually marks showing stages and needs no parameters.
110 -### `snapshot`
112 -```sh
113 -java -jar fxdriver.jar rpc <port> snapshot '{}'
114 -java -jar fxdriver.jar snapshot <port> --summary
115 -```
117 -Optional type filters (exact simple class name, case-sensitive):
119 -```sh
120 -java -jar fxdriver.jar rpc <port> snapshot '{"includeTypes":["Button","TextField"]}'
121 -java -jar fxdriver.jar rpc <port> snapshot '{"excludeTypes":["LabeledText","FontIcon"]}'
122 -```
124 -Both can be combined. `excludeTypes` wins over `includeTypes`. Default (no
125 -filters) returns all visible nodes.
127 -Returns:
129 -- `windows[]`: visible JavaFX windows with bounds and nested nodes.
130 -- `nodes[]`: flat compatibility list.
132 -Important node fields:
134 -- `eid` / `handle`
135 -- `parent`
136 -- `window`
137 -- `type`
138 -- `id`
139 -- `text`
140 -- `accessibleText`, `accessibleHelp`, `accessibleRole`,
141 - `accessibleRoleDescription`
142 -- `styleClasses`
143 -- `visible`, `disabled`, `focused`, `managed`, `mouseTransparent`,
144 - `pickOnBounds`, `opacity`, `cursor`, `actionable`
145 -- `bounds` (screen coordinates), `boundsInLocal`, `layoutBounds`
146 -- optional `semantic` object
148 -Initial `semantic` support:
150 -- `kind`: `labeled`, `button`, `hyperlink`, `toggle`, `checkbox`, `radio`,
151 - `textInput`, `list`, `tree`, `table`, `treeTable`, `comboBox`, `choiceBox`,
152 - `spinner`, `tabPane`, `text`, `control`
153 -- `ComboBox`/`ChoiceBox`: `value`, `itemsCount`, `selectedIndex` (+
154 - `editable`/`showing` for ComboBox); `Spinner`: `value`, `editable`; `TabPane`:
155 - `value` (selected tab text), `tabsCount`, `selectedIndex`
156 -- `Labeled`: `text`, `graphicType`, `contentDisplay`, `mnemonicParsing`,
157 - `wrapText`, `textOverrun`
158 -- `ButtonBase`: `armed`; `Button`: `defaultButton`, `cancelButton`; `Hyperlink`:
159 - `visited`
160 -- `ToggleButton`/`RadioButton`: `selected`, `toggleGroup`
161 -- `CheckBox`: `selected`, `indeterminate`, `allowIndeterminate`
162 -- `TextInputControl`: `value`, `prompt`, `editable`, `selectedText`,
163 - `selectionStart`, `selectionEnd`, `caretPosition`, `anchor`, `length`,
164 - `masked`
165 -- `TextField`: `prefColumnCount`; `TextArea`: `prefColumnCount`, `prefRowCount`,
166 - `wrapText`, `scrollTop`, `scrollLeft`
167 -- `ListView`: `itemsCount`, `selectionMode`, `selectedIndices`, `selectedItems`,
168 - `focusedIndex`, `editable`, `fixedCellSize`, `orientation`, `placeholderText`
169 -- `TreeView`: `itemsCount` (expanded items), `selectionMode`, `selectedIndices`,
170 - `selectedItems`, `focusedIndex`, `editable`, `fixedCellSize`, `rootVisible`,
171 - `rootValue`
172 -- `TableView`: `itemsCount`, `selectionMode`, `selectedIndices`,
173 - `selectedItems`, `focusedIndex`, `focusedCell`, `editable`, `fixedCellSize`,
174 - `placeholderText`, `columns`, `sortOrder`
175 -- `TreeTableView`: `itemsCount` (expanded items), `selectionMode`,
176 - `selectedIndices`, `selectedItems`, `focusedIndex`, `focusedCell`, `editable`,
177 - `fixedCellSize`, `rootVisible`, `rootValue`, `placeholderText`, `columns`,
178 - `sortOrder`
180 -`PasswordField` text/value/selectedText are redacted by default
181 -(`masked: true`). Virtualized controls expose model/selection metadata;
182 -unrealized rows/cells are not emitted as nodes.
184 -### Snapshot summary
186 -```sh
187 -java -jar fxdriver.jar snapshot <port> --summary
188 -```
190 -The CLI summary command returns a compact first view. It caps windows at 8,
191 -buttons at 24, text fields at 16, value controls at 24, menus at 16, tables at
192 -12, selected tabs at 12, and the distinct visible-text sample at 40 strings. Integration tests enforce a 16 KiB
193 -response bound for the focused fixture.
195 -- `windows[]`: window id/title/focus/bounds.
196 -- `buttons[]`: visible `ButtonBase` controls with text and disabled state.
197 -- `textFields[]`: text inputs with value/prompt/editability.
198 -- `controls[]`: toggles and choice/date/spinner/range values.
199 -- `menus[]`: menu controls and context-menu targets with nested item paths.
200 -- `tables[]`: visible `TableView` and `TreeTableView` row/column summaries.
201 -- `selectedTabs[]`: selected tab text per visible `TabPane`.
202 -- `focusedNode`: target metadata or `null`.
203 -- `visibleTextSample[]`: bounded visible text for orientation.
205 -Use full `snapshot` when you need complete semantics or raw node hierarchy.
207 -### `highlight`
209 -```sh
210 -java -jar fxdriver.jar rpc <port> highlight '{"selector":"type=Button"}'
211 -java -jar fxdriver.jar rpc <port> highlight '{"selector":"type=Button","effect":"invert"}'
212 -java -jar fxdriver.jar rpc <port> highlight '{"selector":"type=Button","effect":"glow"}'
213 -```
215 -Applies a temporary non-layout JavaFX `Effect` to raw matching `Node`s,
216 -including non-control nodes such as `Text`, layout panes, and `SubScene`.
217 -Default `invert` effect uses blend-difference against white over the node's
218 -local bounds, so it is visible on both light and dark themes without changing
219 -layout. Use to verify selector ambiguity or show humans what target will be
220 -used. Calling `highlight` with no matches clears prior highlights. Visual trace
221 -for actions highlights the resolved actionable target instead.
223 -### `click`
225 -```sh
226 -java -jar fxdriver.jar rpc <port> click '{"nodeId":"find-duplicates"}'
227 -java -jar fxdriver.jar rpc <port> click '{"textExact":"Keep duplicates"}'
228 -java -jar fxdriver.jar rpc <port> click '{"textExact":"Projects","untilQuietMs":500,"timeoutMs":10000}'
229 -```
231 -Fires `ButtonBase` directly when possible; otherwise uses JavaFX Robot center
232 -click. With `untilQuietMs`, waits after the click until the visible UI signature
233 -has stayed stable for that many milliseconds, or returns `QUIET_TIMEOUT`.
235 -### `fire`
237 -```sh
238 -java -jar fxdriver.jar rpc <port> fire '{"textExact":"Find duplicates..."}'
239 -```
241 -Calls `ButtonBase.fire()` on first matching button-like control.
243 -### `type`, `setText`, `clear`
245 -```sh
246 -java -jar fxdriver.jar rpc <port> setText '{"nodeId":"filter-duplicates","value":"portrait"}'
247 -java -jar fxdriver.jar rpc <port> clear '{"nodeId":"filter-duplicates"}'
248 -java -jar fxdriver.jar rpc <port> type '{"nodeId":"filter-duplicates","value":"abc","replace":false}'
249 -```
251 -Prefer `setText` for deterministic tests; use `type` when simulating user input
252 -matters. These set the control's text directly; for real keystrokes (Enter to
253 -submit a dialog, Tab to move focus, accelerators), use `key`.
255 -### `key`
257 -```sh
258 -java -jar fxdriver.jar rpc <port> key '{"key":"ENTER"}'
259 -java -jar fxdriver.jar rpc <port> key '{"key":"CTRL+S"}'
260 -java -jar fxdriver.jar rpc <port> key '{"nodeId":"search","chars":"hello"}'
261 -```
263 -Dispatches synthetic JavaFX `KeyEvent`s to a node; it is not OS/global input.
264 -`key` accepts one named key or chord such as `ENTER`, `ESCAPE`/`ESC`, `TAB`,
265 -`F1`, `CTRL+S`, `SHIFT+TAB`, or `META+K`. `chars` emits one `KEY_TYPED` event
266 -per Unicode code point. With a selector (`nodeId`, `textExact`, …), the first
267 -match is focused; otherwise the current JavaFX focus owner is used. The result
268 -contains `key`, `chars`, `sent`, and `target`; no target returns `NO_FOCUS`.
270 -### `setValue`, `selectIndex`, `selectListItem`
272 -```sh
273 -java -jar fxdriver.jar rpc <port> setValue '{"nodeId":"country","value":"Germany"}'
274 -java -jar fxdriver.jar rpc <port> selectIndex '{"nodeId":"projects","index":2}'
275 -java -jar fxdriver.jar rpc <port> selectListItem '{"nodeId":"duplicates-list","itemText":"portrait"}'
276 -```
278 -Drives controls that `click`/`fire` cannot reliably set. Use `setValue` for
279 -controls with a direct value such as `ChoiceBox`, `ComboBox`, `DatePicker`,
280 -`Slider`, and text-like value controls. Use `selectIndex` for indexed selection
281 -in list/table/tree-style controls. Use `selectListItem` for `ListView` by item
282 -text. A generic `select` method is intentionally unsupported.
284 -### Choice, popup, and step controls
286 -```sh
287 -java -jar fxdriver.jar rpc <port> setValue '{"nodeId":"theme","value":"Dark"}'
288 -java -jar fxdriver.jar rpc <port> showPopup '{"nodeId":"density"}'
289 -java -jar fxdriver.jar rpc <port> hidePopup '{"nodeId":"density"}'
290 -java -jar fxdriver.jar rpc <port> increment '{"nodeId":"days","steps":5}'
291 -java -jar fxdriver.jar rpc <port> decrement '{"nodeId":"days","steps":2}'
292 -```
294 -`setValue` supports choice/combo/date/range/spinner controls. Increment and
295 -decrement default to one step.
297 -### `tableCell`
299 -```sh
300 -java -jar fxdriver.jar rpc <port> tableCell '{"nodeId":"projects","tableText":"MDP_40982497","column":"Projektnummer"}'
301 -java -jar fxdriver.jar rpc <port> tableCell '{"nodeId":"projects","rowIndex":2,"columnIndex":0}'
302 -```
304 -Reads `TableView` or `TreeTableView` model values by row text/row index and
305 -flat top-level column text, id, or index. It does not depend on realized cell
306 -nodes. The result includes `rowIndex`, `columnIndex`, resolved `column`, full
307 -`rowText`, `value`, and `ok`; failures use `NO_TABLE`, `NO_ROW`, or `NO_COLUMN`.
309 -### `fireMenuItem`
311 -```sh
312 -java -jar fxdriver.jar rpc <port> fireMenuItem '{"nodeId":"main-menu","path":["File","Save"]}'
313 -java -jar fxdriver.jar rpc <port> fireMenuItem '{"nodeId":"actions","itemText":"Delete"}'
314 -```
316 -Fires `MenuBar`, `MenuButton`, `SplitMenuButton`, or context-menu items without
317 -showing a JavaFX popup. It cannot operate native menus or dialogs. Use `path`
318 -for nested traversal or `itemText` for exact-then-contains recursive lookup.
319 -The result contains `query`, `kind`, `path`, `value`, and `fired`; failures use
320 -`NO_MENU_ITEM` or `DISABLED`.
322 -### `wait`
324 -```sh
325 -java -jar fxdriver.jar rpc <port> wait '{"text":"Done","timeoutMs":5000}'
326 -java -jar fxdriver.jar rpc <port> wait '{"nodeId":"save","enabled":true,"timeoutMs":5000}'
327 -```
329 -Polls until selector is present by default. Default timeout is 2000 ms; pass
330 -explicit `timeoutMs` for slow app operations. Optional predicates: `present`,
331 -`enabled`, `focused`, `visible`. Like the action commands, `wait` and `assert`
332 -only consider visible nodes unless `visible:false` is passed explicitly, so a
333 -satisfied `wait` means the follow-up `click`/`type` sees the same node. Failed
334 -waits return `ok:false` with `TIMEOUT` and up to five bounded `nearMatches`.
335 -Each candidate has `eid`, `type`, `id`, `text`, and diagnostic `selectorPath`.
336 -Near matches and selector paths explain misses; they are not supported
337 -selectors. Successful waits omit `nearMatches`.
339 -### `assert`
341 -```sh
342 -java -jar fxdriver.jar rpc <port> assert '{"textExact":"Keep duplicates","present":true}'
343 -java -jar fxdriver.jar rpc <port> assert '{"selector":"type=Button","count":3}'
344 -```
346 -Returns `{ok,count}`. The CLI exits non-zero only on a protocol-level JSON-RPC
347 -error (top-level `error` member); app-level `ok:false` results exit zero and
348 -need inspection in scripts.
350 -### `events`, `eventsSince`, `clearEvents`
352 -```sh
353 -java -jar fxdriver.jar rpc <port> events '{}'
354 -java -jar fxdriver.jar rpc <port> eventsSince '{"cursor":20}'
355 -java -jar fxdriver.jar rpc <port> eventsSince '{"since":1735689600000}'
356 -java -jar fxdriver.jar rpc <port> events '{"clear":true}'
357 -java -jar fxdriver.jar rpc <port> clearEvents '{}'
358 -```
360 -Returns recent fxdriver calls with `ts`, `method`, `ok`, and `params`. If
361 -request params include `tag` or `scenario`, the event copies them to top-level
362 -fields. `eventsSince` accepts an event cursor or millisecond timestamp. Use `events {"clear":true}` to read and clear in one call.
364 -### `screenshot`
366 -CLI shortcut for the focused showing stage, then first showing stage:
368 -```sh
369 -java -jar fxdriver.jar screenshot <port> target/fxdriver.png
370 -```
372 -RPC supports window/default, node, and rect targets:
374 -```sh
375 -java -jar fxdriver.jar rpc <port> screenshot '{"path":"/tmp/run/window.png"}'
376 -java -jar fxdriver.jar rpc <port> screenshot '{"target":"node","nodeId":"save","path":"/tmp/run/save.png"}'
377 -java -jar fxdriver.jar rpc <port> screenshot '{"target":"rect","x":0,"y":0,"w":320,"h":200,"path":"/tmp/run/crop.png"}'
378 -java -jar fxdriver.jar rpc <port> screenshot '{"path":"/tmp/run/dialog.png","window":"w2"}'
379 -```
381 -A relative `path` resolves against the target app's working directory, not the
382 -caller's — pass an absolute path (or use the CLI `screenshot` subcommand, which
383 -absolutizes before sending). The result echoes the absolute path written. Window
384 -and rect targets default to the focused showing JavaFX stage, then the first
385 -showing stage; pass `window` with a stage `wid` from snapshot (`"w2"`) to
386 -capture that exact stage. Popup window ids are rejected. `press` accepts the
387 -same parameter to scope accelerators.
389 -Captures optimized lossless PNG. RPC results include `path`, `bytes`, `target`,
390 -`image` bounds, `source` bounds/units, `scale`, `activeWindow`, and a bounded
391 -`visibleTextSample` for orientation. `activeWindow` contains only id, title,
392 -bounds, and focused-node metadata—not a recursive snapshot. CLI screenshot
393 -prints that result plus image summary. Opaque screenshots may be encoded as
394 -RGB; transparent screenshots preserve alpha.
396 -### Video and modal dispatch
398 -`videoStart`, `videoStep`, and `videoStop` are not available in this phase.
399 -Modal-safe asynchronous action dispatch is also deferred. Check `capabilities`
400 -rather than assuming either API exists.
402 -### `image-summary`
404 -```sh
405 -java -jar fxdriver.jar image-summary target/fxdriver.png
406 -```
408 -Reports path, bytes, width, height, average color, and average hash.
410 -### `image-diff`
412 -```sh
413 -java -jar fxdriver.jar image-diff before.png after.png target/diff.png
414 -```
416 -Reports changed pixels, percent, changed bounds, size mismatch, and optional
417 -optimized lossless diff PNG path.
419 -### `listItems`
421 -```sh
422 -java -jar fxdriver.jar rpc <port> listItems '{"nodeId":"duplicates-list"}'
423 -```
425 -Returns `ListView` item strings.
427 -### `selectListItem`
429 -```sh
430 -java -jar fxdriver.jar rpc <port> selectListItem '{"nodeId":"duplicates-list","itemText":"portrait"}'
431 -```
433 -Selects matching item. Default `activate=true` also fires a matching realized
434 -row button if available.
436 -### `scroll`
438 -```sh
439 -java -jar fxdriver.jar rpc <port> scroll '{"selector":"type=ScrollPane","amount":6}'
440 -```
442 -Uses control-specific scroll when possible; Robot fallback otherwise.
444 -### `shutdown`
446 -```sh
447 -java -jar fxdriver.jar rpc <port> shutdown '{}'
448 -```
450 -Stops the RPC server. Apps started by `launch` exit; attached apps keep running. `exitApp:true` explicitly exits either.
452 -## Selectors
454 -Supported forms:
456 -```text
457 -@handle capability label for numeric handles
458 -@42 handle
459 -eid=n42 node eid
460 -#node-id JavaFX node id
461 -.style-class JavaFX style class
462 -=exact text exact visible/control text
463 -text=Open contains visible/control text
464 -regexText=Open.* regex over visible/control text
465 -type=Button Java class simple name
466 -role=BUTTON accessible role
467 -accessible=Save accessible text contains
468 -```
470 -JSON params that map to selectors:
472 -```json
473 -{ "handle": 42 }
474 -{ "nodeId": "save" }
475 -{ "cssId": "save" }
476 -{ "styleClass": "primary" }
477 -{ "textExact": "Save" }
478 -{ "text": "Save" }
479 -{ "regexText": "Sa.*" }
480 -{ "type": "Button" }
481 -{ "role": "BUTTON" }
482 -{ "accessible": "Save" }
483 -{ "selector": "type=Button" }
484 -```
486 -Snapshot and near-match `selectorPath` values are diagnostics only and cannot
487 -be passed back as selectors. Generic `select` is unsupported; use the explicit
488 -selection methods above.

pi/agent/skills/fxdriver/references/troubleshooting.md

Deleted
deleted file mode 100644
1 -# troubleshooting fxdriver
3 -## Attach fails
5 -Check PID:
7 -```sh
8 -jcmd -l
9 -```
11 -Common causes:
13 -- process exited;
14 -- wrong PID;
15 -- target is not a compatible JVM;
16 -- dynamic attach restricted;
17 -- target runtime image lacks attach support;
18 -- port already in use.
20 -For the current prototype, launching the app with debug flags has worked:
22 -```sh
23 --XX:+EnableDynamicAgentLoading
24 ---add-modules=java.instrument
25 -```
27 -Do not assume final distribution will require these; integration remains open.
29 -## RPC doesn't respond
31 -Check attach output for endpoint/port.
33 -Try:
35 -```sh
36 -java -jar target/fxdriver.jar rpc <port> ping '{}'
37 -```
39 -If port is closed, reattach or inspect app stderr.
41 -## Selector matches too much
43 -Text queries may match both a control and skin/internal text. Prefer:
45 -```json
46 -{ "nodeId": "save" }
47 -{ "textExact": "Save" }
48 -{ "selector": "role=BUTTON" }
49 -```
51 -Use highlight:
53 -```sh
54 -java -jar target/fxdriver.jar rpc <port> highlight '{"text":"Save"}'
55 -```
57 -## Click doesn't do expected thing
59 -Prefer semantic/direct actions when available:
61 -- inspect `semantic.kind`/state before acting;
62 -- `fire` for `ButtonBase`;
63 -- `setText` for `TextInputControl`;
64 -- `selectListItem` for `ListView`.
66 -Robot click can be backend/session sensitive, especially Wayland.
68 -## Text/password fields
70 -`PasswordField` snapshot text/value is redacted by default (`masked: true`). Use app-specific test hooks if tests must verify actual secrets; do not expect password values in snapshots.
72 -## Virtualized controls
74 -`ListView`, `TreeView`, `TableView`, and `TreeTableView` only realize visible cells. Snapshot shows visible nodes, not all model rows. Use `semantic` metadata for counts, selected indices/items, focus, root value, and table columns.
76 -Use model primitives where available:
78 -```sh
79 -java -jar target/fxdriver.jar rpc <port> listItems '{"nodeId":"my-list"}'
80 -java -jar target/fxdriver.jar rpc <port> selectListItem '{"nodeId":"my-list","itemText":"needle"}'
81 -```
83 -## Visual bugs
85 -JSON assertions can pass while layout is visually wrong. Capture screenshots:
87 -```sh
88 -java -jar target/fxdriver.jar screenshot <port> target/failure.png
89 -```
91 -Read the screenshot. Use `image-diff` to localize changes, not to decide quality alone.
93 -## Native dialogs
95 -Native OS file/color dialogs are outside JavaFX scene graph. They are not MVP. Prefer app test mode or avoid native dialogs in fxdriver flows for now.
97 -## WebView
99 -MVP treats `WebView` as a JavaFX node with bounds. DOM-level automation is later.

pi/agent/skills/fxdriver/references/visual.md

Deleted
deleted file mode 100644
1 -# visual artifacts
3 -fxdriver visual primitives are inputs for agent judgment, not a built-in quality
4 -policy.
6 -## Screenshot
8 -```sh
9 -java -jar fxdriver.jar screenshot <port> target/ui.png
10 -```
12 -Output shape:
14 -```json
16 - "screenshot": {
17 - "jsonrpc": "2.0",
18 - "result": {
19 - "path": "...",
20 - "bytes": 12345,
21 - "activeWindow": { "title": "Main" },
22 - "visibleTextSample": ["Search", "Save"]
23 - }
24 - },
25 - "summary": {
26 - "path": "...",
27 - "bytes": 12345,
28 - "width": 1256,
29 - "height": 1408,
30 - "avgColor": { "r": 241, "g": 239, "b": 232, "a": 255 },
31 - "ahash": "feffdfdf7f7f7f3b"
32 - }
34 -```
36 -Read the image when making visual/layout/color claims. Screenshots are optimized
37 -lossless PNGs by default (max PNG compression where available, RGB when fully
38 -opaque). To avoid context bloat, prefer the smallest screenshot that can answer
39 -the question once node/rect screenshots exist; keep full-size PNGs on disk for
40 -audit/diff truth.
42 -## Image summary
44 -```sh
45 -java -jar fxdriver.jar image-summary target/ui.png
46 -```
48 -Use for quick sanity:
50 -- dimensions non-zero;
51 -- file size plausible;
52 -- hash changed/didn't change;
53 -- average color roughly expected.
55 -Do not use average color/hash as proof of layout quality.
57 -## Image diff
59 -```sh
60 -java -jar fxdriver.jar image-diff target/before.png target/after.png target/diff.png
61 -```
63 -Output fields:
65 -- `changedPixels`
66 -- `changedPercent`
67 -- `bounds`
68 -- `sizeMismatch`
69 -- `diff` path if requested
71 -Use diff to find if something changed and where. Then inspect `after.png` and/or
72 -`diff.png`. Diffs are meaningful only when images share compatible source rect,
73 -scale, and dimensions.
75 -## Context-efficient visual workflow
77 -Use this progressive order:
79 -1. `snapshot --summary` for windows/titles/key controls.
80 -2. Full `snapshot` for structure/bounds/semantics.
81 -3. `image-summary` for dimensions/hash sanity.
82 -4. Read full screenshot only when layout/color/visual quality matters.
83 -5. Prefer cropped/node/rect screenshots when they answer the question.
84 -6. If downscaled/cropped image is unclear, request a larger crop or full PNG.
86 -Screenshot target examples:
88 -```json
89 -{"target":"node","selector":"#details","path":"target/details.png"}
90 -{"target":"rect","window":"w1","x":0,"y":0,"w":640,"h":360,"path":"target/crop.png"}
91 -```
93 -Do not use downscaled/cropped images as sole proof for tiny text, contrast, or
94 -pixel-perfect diffs. JPEG/WebP output is intentionally not planned for now; UI
95 -screenshots stay PNG.
97 -## Common visual workflow
99 -```sh
100 -java -jar fxdriver.jar screenshot <port> target/before.png
101 -# make app/code change; rebuild/relaunch/act
102 -java -jar fxdriver.jar screenshot <port> target/after.png
103 -java -jar fxdriver.jar image-diff target/before.png target/after.png target/diff.png
104 -```
106 -Checklist for human-readable UI:
108 -- text contrast readable;
109 -- icons visible on button backgrounds;
110 -- no important text clipped/truncated unexpectedly;
111 -- no controls off-screen;
112 -- toolbars wrap or scroll instead of overflowing;
113 -- panels have reasonable spacing;
114 -- selected/disabled/focus states are distinguishable;
115 -- screenshots match JSON state.
117 -fxdriver should expose primitives for this; agent/skill decides if pass/fail.

pi/agent/skills/fxdriver/references/workflow.md

Deleted
deleted file mode 100644
1 -# fxdriver workflow
3 -## Attach
5 -Find Java processes:
7 -```sh
8 -jcmd -l
9 -```
11 -Attach:
13 -```sh
14 -java -jar fxdriver.jar attach <pid>
15 -export FXDRIVER_TOKEN=<printed-token>
16 -java -jar fxdriver.jar rpc <printed-port> ping '{}'
17 -```
19 -## Observe
21 -Start with the compact summary when orienting:
23 -```sh
24 -java -jar fxdriver.jar snapshot <printed-port> --summary > target/fxdriver-summary.json
25 -```
27 -Use a full snapshot when the summary is not enough:
29 -```sh
30 -java -jar fxdriver.jar rpc <printed-port> snapshot '{}' > target/fxdriver-snapshot.json
31 -```
33 -Look for:
35 -- visible windows;
36 -- stable `nodeId` values;
37 -- actionable buttons/text inputs/lists;
38 -- exact text vs duplicate skin/internal text;
39 -- disabled/focused/common node state;
40 -- `semantic.kind` and control-specific state (`selected`, `value`, `prompt`,
41 - `editable`, checkbox indeterminate state, radio toggle group, list/table
42 - counts, selected indices/items, focused index, columns, etc.).
44 -## Choose selectors
46 -Preference order:
48 -1. `nodeId` / `#id`
49 -2. `eid` if stable within this attached session
50 -3. `textExact`
51 -4. `role` + text/accessibility
52 -5. `type` for broad discovery only
53 -6. bare text contains only as last resort
55 -Do not copy snapshot or near-match `selectorPath` diagnostics into requests.
56 -They are deliberately not selectors.
58 -Verify ambiguous selectors with `highlight`:
60 -```sh
61 -java -jar fxdriver.jar rpc <port> highlight '{"text":"Find"}'
62 -```
64 -If the workflow needs repeated or future automation and important controls lack
65 -stable ids, ask the human for permission to add JavaFX ids to the app under
66 -test. When approved, add the smallest useful set of semantic ids, follow local
67 -style and naming conventions, and leave them in the app. Do not add broad,
68 -mechanical ids to every node; target controls that represent user intent,
69 -navigation landmarks, form fields, menus, tables/lists, and primary actions.
71 -## Act
73 -Examples:
75 -```sh
76 -java -jar fxdriver.jar rpc <port> click '{"nodeId":"find-duplicates"}'
77 -java -jar fxdriver.jar rpc <port> click '{"textExact":"Projects","untilQuietMs":500,"timeoutMs":10000}'
78 -java -jar fxdriver.jar rpc <port> setText '{"nodeId":"filter-duplicates","value":"portrait"}'
79 -java -jar fxdriver.jar rpc <port> selectListItem '{"nodeId":"duplicates-list","itemText":"portrait"}'
80 -java -jar fxdriver.jar rpc <port> key '{"nodeId":"filter-duplicates","key":"ENTER"}'
81 -java -jar fxdriver.jar rpc <port> fireMenuItem '{"nodeId":"main-menu","path":["File","Save"]}'
82 -java -jar fxdriver.jar rpc <port> tableCell '{"nodeId":"projects","tableText":"MDP_40982497","column":"Projektnummer"}'
83 -```
85 -`key` is synthetic JavaFX input, not OS/global input. Menu firing resolves and
86 -fires the model item without showing its JavaFX popup. `tableCell` reads model
87 -values from flat columns. There is no generic `select`; choose the explicit
88 -selection method for the control.
90 -## Wait/assert
92 -After action, wait for observable state:
94 -```sh
95 -java -jar fxdriver.jar rpc <port> wait '{"text":"portrait","timeoutMs":30000}'
96 -java -jar fxdriver.jar rpc <port> assert '{"textExact":"Keep duplicates","present":true}'
97 -```
99 -Do not assume immediate UI update after action.
101 -If a wait fails, read the returned `ok:false`/`TIMEOUT` and its at-most-five
102 -`nearMatches`, then take a fresh summary or full snapshot if needed. Near-match
103 -ids/text/types help diagnose a typo; their selector paths are not selectors.
105 -## Visual review
107 -When task mentions colors, layout, overflow, visual quality, screenshots are
108 -mandatory:
110 -```sh
111 -java -jar fxdriver.jar screenshot <port> target/before.png
112 -# act/change app
113 -java -jar fxdriver.jar screenshot <port> target/after.png
114 -java -jar fxdriver.jar image-diff target/before.png target/after.png target/diff.png
115 -```
117 -Read screenshots. Diff alone tells change amount, not quality. Screenshot JSON
118 -also identifies `activeWindow` and provides a bounded `visibleTextSample` for
119 -orientation; it is not a replacement for reading the image.
121 -## Video evidence
123 -Video RPCs are not available in this phase. Check `capabilities`; use repeated
124 -screenshots when still-image evidence is sufficient. Modal-safe asynchronous
125 -action dispatch is also outside this phase.
127 -## Failure artifacts
129 -On failure, collect:
131 -```sh
132 -java -jar fxdriver.jar rpc <port> snapshot '{}' > target/failure-snapshot.json
133 -java -jar fxdriver.jar screenshot <port> target/failure.png > target/failure-summary.json
134 -java -jar fxdriver.jar rpc <port> events '{"clear":true}' > target/failure-events.json
135 -```
137 -Then inspect app logs separately.
139 -## Clean up
141 -```sh
142 -java -jar fxdriver.jar rpc <port> shutdown '{}'
143 -```
145 -This stops only fxdriver RPC server. Kill the app process separately if the run
146 -launched it.