Luigit
repositories / smith

smith

There are many coding harnesses - but this one is fast

owned by admin

.system/issues/SMH-ISSUE-FRXNMTWL-nushell-dependency-reuse-for-independent-sdk-fac/index.md

Raw
Rendered preview

id: SMH-ISSUE-FRXNMTWL type: issue title: "Nushell Dependency Reuse for Independent SDK Facilities" specs: [SMH-SPEC-SPEC0001, SMH-SPEC-ZVMD_XYT, SMH-SPEC-PLUG0001]

Nushell Dependency Reuse for Independent SDK Facilities

Symptom

Nushell brings useful underlying libraries, but Smith lacks a complete evaluated inventory separating reusable SDK facilities from interpreter internals and optional Nu plugins. Plugins should access suitable facilities directly, without Nu scripts, evaluation grants, or Nu types. Dependency overlap is an opportunity, not an adoption decision.

Impact

Unevaluated reuse risks redundant interpretation or native engine duplication. Blind reuse expands dependencies, capabilities, build costs, and runtime budgets without proven consumers. Optional upstream features can be mistaken for libraries already paid for by Smith.

Evidence scope

Desk investigation sampled Nushell release 0.116.0; this is not an approved Smith version. Reviewed command, utility, format/query/dataframe/Git plugin, and optional tooling manifests; selected command implementations. No builds, cost measurements, full transitive resolution, or SDK prototypes performed.

Current Cargo.lock SHA-256: bfa8b0131ef232ee6c15fa4757bb63f7d572daf73c5f6a833f9c69c832d381e3. No Nushell, notify, rusqlite, Polars, or ndarray packages are present. Current workspace dependencies already sanction JSON, HTTP/TLS, search, and terminal stacks. Lock membership includes development/build dependencies; it does not establish shipped-runtime availability or sanctioned direct ownership.

Verified distinctions

  • Watch: notify-debouncer-full 0.7.0, enabled by nu-command/os, with a transitive notify backend.1
  • SQLite: rusqlite 0.40.2, optional nu-command/sqlite, with bundled SQLite C plus backup, chrono, and column metadata features.1
  • Dataframes: Polars 0.55.2 belongs to separate nu_plugin_polars, not the parser/evaluator baseline; its manifest includes cloud providers, AWS credential machinery, and Tokio.2
  • YAML: nu-heavy-utils/yaml enables serde-saphyr; the helper crate itself depends on Nu protocol types.3
  • Native SQLite C allocations are not automatically covered by Smith's Rust global allocator; accounting requires verification.

Candidate inventory

Candidate verdicts are first-pass judgments, not approved interfaces. Unmeasured costs and missing consumers remain open for every new facility.

Feature family Underlying libraries Sample graph position Independent SDK opportunity / review
File watch notify-debouncer-full, notify Optional os command feature Strong candidate: granted roots, pull batches, overflow/rescan semantics, recorded event ordering, cancellation.
SQLite rusqlite, libsqlite3-sys Optional sqlite feature; bundled native C Candidate plugin-local storage/query, not replacement session format; SQL authorization, transactions, interruption, row/byte limits, native allocation accounting.
Dataframes Polars family, sqlparser Separate Nu plugin Conditional candidate: bounded columnar batches and query resources; isolate compute from filesystem/cloud access; investigate minimal feature set before adoption.
Numeric arrays ndarray Direct command dependency Conditional numeric batch operations; establish consumers, shape semantics, allocation bounds, and boundary-copy costs.
Delimited records csv Direct command dependency Candidate bounded streaming record parser; parsing does not imply filesystem access.
Spreadsheets calamine Direct command dependency Conditional sheet readers; expansion limits, type/date mapping, complete versus streaming access.
XML roxmltree, quick-xml Direct command dependencies Conditional tree/query or streaming parser; entity policy, depth bounds, one selected owner per concern.
JSON / Nu formats serde_json, nu-json, nuon Command dependencies Reuse Smith JSON stack; Nu-specific protocol coupling is not an independent SDK contract.
YAML serde-saphyr Heavy-utils YAML feature Conditional parser; explicit tags, aliases, scalar resolution, limits; do not expose Nu helper types.
TOML / KDL toml, toml_edit, kdl Direct command dependencies Conditional parse/edit operations; existing TOML lock presence is not an application facade.
MessagePack rmp Direct command dependency Conditional codec; do not replace or duplicate Smith's typed-session/CBOR contract.
HTML / JSON query / XPath scraper, gjson, sxd-document, sxd-xpath, webpage Separate query plugin Conditional extraction/query; pure parsing separate from network grants; avoid another HTTP stack.
Mail / calendar / INI / plist eml-parser, ical, rust-ini, plist Separate formats plugin Format-specific candidates only with actual consumers; not evaluator-inherent dependencies.
Markdown query mq-markdown Direct command dependency Potential docs extraction/query; inspect behavior, limits, and existing documentation needs before adding an engine.
Search / fuzzy / regex fff-search, nucleo-matcher, fancy-regex Commands/utils; fff-search under os Reuse existing ignore search owner first; distinct fuzzy/regex operations need explicit semantics and bounded work.
Compression brotli; transport gzip through ureq features Commands / optional network feature Conditional bounded byte-stream codec; expansion limits, cancellation, no extra transport owner.
Hashes / byte encodings digest, sha2, md-5, base64, data-encoding Direct command dependencies Candidate pure bounded transforms; graph versions differ; MD5 is not a security-integrity primitive.
Text encodings encoding_rs, chardetng Direct command dependencies Candidate explicit decode/detect operations; detection uncertainty and invalid-byte policy must remain visible.
Dates / time zones chrono, chrono-tz, dtparse, human-date-parser Direct command dependencies Conditional pure parse/format; pinned locale/timezone semantics; no ambient current time.
Glob / path / timestamps nu-glob, wax, filetime, uucore, uu_* Commands/system; several os gates Existing file/search mechanisms first; investigate only gaps, symlink scopes, matching semantics, mutation evidence.
Host / process telemetry sysinfo, procfs, platform helpers Commands/system, target-specific Conditional narrow telemetry; sensitive data grants, platform vocabulary, recorded nondeterministic input.
HTTP / TLS / DNS ureq, rustls, dns-lookup Optional network/TLS features Reuse existing smith-harness HTTP engine; independent DNS only for demonstrated need.
Git metadata git2 Separate nu_plugin_gstat Conditional repository inspection; native build and repository scope review; no automatic second VCS implementation.
Clipboard / trash / launch arboard, trash, open Target-specific or optional OS features Product-specific granted effects, not automatic SDK inventory; privacy, destructive behavior, supervision.
LSP / DAP / MCP lsp-server, lsp-types, rmcp, schemars; Nu adapters Optional top-level tooling components Protocol/library reuse only when frontend requirements justify it; Nu adapters are not generic SDK APIs; MCP adds async/HTTP machinery.
Terminal/editor support ratatui, crossterm, reedline, edtui, table/color helpers CLI/pager components Reuse existing terminal/UI vocabulary; do not expose engine widgets or pull an interactive Nu shell into the SDK.
Internal/support helpers Nu protocol/parser utilities, macro, string, locale, sorting, MIME, endian/merge helpers Mixed command/utility dependencies Review individual pure primitives if needed; no one-to-one SDK API per dependency, no automatic public Nu helper layer.

Existing lock overlaps

  • HTTP: ureq 3.4.2, rustls 0.23.45; existing engine owner.
  • JSON/UI: serde_json 1.0.151, ratatui 0.30.2, crossterm 0.29.0; existing sanctioned concerns.
  • TOML: toml 0.8.23 through cargo-util-schemas, toml_edit 0.22.27 through TOML.
  • Regex: fancy-regex 0.11.0 through termwiz; Nu sample requests 0.19.
  • Hashes: sha2 0.10.9 and digest 0.10.7 through Wasmtime/terminal dependencies; Nu sample requests 0.11 lines.
  • Encoding: encoding_rs 0.8.42 through Wasmtime.
  • Base64: 0.22.1 and 0.23.1 through terminal/HTTP dependencies; reconcile exact features and versions before claiming shared code.

Presence alone proves neither zero incremental cost nor a reusable public API.

Target relationship

Independent SDK access reuses a library, not Nushell evaluation.

flowchart TB
  sdk["Plugin SDK"] --> facade["Host facade"]
  facade --> library["Shared crate"]
  nu["Nu"] --> library

The actual selected feature graph determines whether the crate is shared or newly introduced.

Reproduction / continuation

  1. Resolve the selected Nushell embedding crates/features, separately from CLI and optional plugin graphs.
  2. Map each reachable feature to its underlying library and actual command callsites; classify optional additions explicitly.
  3. Compare resolved target/runtime versions and features with Smith ownership; exclude dev/build-only overlap.
  4. For every candidate, identify two real consumers and independently useful Smith operations, or record rejection/defer rationale.
  5. Inspect license/advisory/native-build/musl support, grants, determinism, recording, replay, cancellation, and bounds.
  6. Measure incremental build size/time, startup, resident memory, allocations, and native-versus-Nu/WIT invocation costs under matched conditions before claiming a benefit.
  7. Promote only proven needs into spec work; no automatic public API or dependency adoption.

Any adopted library becomes an explicit sanctioned workspace dependency behind one owner/facade, with Smith types at boundaries. Existing HTTP, search, JSON, session codec, and UI owners remain authoritative. No new application Cargo features or interpreter dependency for independent SDK consumers.

Evidence limitations

This covers feature families from sampled command/utility and named plugin/tooling manifests, not every transitive crate or Nushell command. Implementation-to-library mapping was verified for watch, SQLite, selected format readers, and hashes; other rows retain manifest-level evidence. Full target-specific resolution, license review of every candidate, real consumers, native prototypes, musl builds, allocation accounting, and measurements remain outstanding. No specs, code, dependencies, or budget pins changed by this investigation.

Completion criteria

  • Selected embedding graph and optional additions have a complete classified inventory.
  • Every candidate has verified callsites, graph membership, consumers, compatibility/cost evidence, and an adopt/reject/defer rationale.
  • Proposed SDK facilities are independent of Nu evaluation and respect existing effect and world-shape contracts.

Related contracts

  • Smith Specification: reusable native SDK facilities, plugin-owned tools, durable effects.
  • Embedded Nushell: evaluator parity and effect safety; no claim that optional plugins belong to the evaluator.
  • Plugin System: granted interfaces, Smith vocabulary, pull/batch semantics, two real consumers.

Selected command evidence: CSV reader, XML reader, SHA-256.

Additional manifest evidence: extra commands, utilities, formats plugin, query plugin, Git plugin, system support, LSP, DAP, MCP, pager/editor.

  1. Nu command dependencies/features, workspace versions, watch implementation, SQLite implementation. ↩ ↩2

  2. Dataframe plugin dependencies/features, plugin registration. ↩

  3. Heavy-utils YAML dependency/features. ↩

---
id: SMH-ISSUE-FRXNMTWL
type: issue
title: "Nushell Dependency Reuse for Independent SDK Facilities"
specs: [SMH-SPEC-SPEC0001, SMH-SPEC-ZVMD_XYT, SMH-SPEC-PLUG0001]
---

# Nushell Dependency Reuse for Independent SDK Facilities

## Symptom

Nushell brings useful underlying libraries, but Smith lacks a complete evaluated inventory separating reusable SDK facilities from interpreter internals and optional Nu plugins.
Plugins should access suitable facilities directly, without Nu scripts, evaluation grants, or Nu types.
Dependency overlap is an opportunity, not an adoption decision.

## Impact

Unevaluated reuse risks redundant interpretation or native engine duplication.
Blind reuse expands dependencies, capabilities, build costs, and runtime budgets without proven consumers.
Optional upstream features can be mistaken for libraries already paid for by Smith.

## Evidence scope

Desk investigation sampled Nushell release 0.116.0; this is not an approved Smith version.
Reviewed command, utility, format/query/dataframe/Git plugin, and optional tooling manifests; selected command implementations.
No builds, cost measurements, full transitive resolution, or SDK prototypes performed.

Current [Cargo.lock](../../../Cargo.lock) SHA-256: `bfa8b0131ef232ee6c15fa4757bb63f7d572daf73c5f6a833f9c69c832d381e3`.
No Nushell, notify, rusqlite, Polars, or ndarray packages are present.
Current [workspace dependencies](../../../Cargo.toml) already sanction JSON, HTTP/TLS, search, and terminal stacks.
Lock membership includes development/build dependencies; it does not establish shipped-runtime availability or sanctioned direct ownership.

### Verified distinctions

- Watch: `notify-debouncer-full` 0.7.0, enabled by `nu-command/os`, with a transitive notify backend.[^commands]
- SQLite: `rusqlite` 0.40.2, optional `nu-command/sqlite`, with bundled SQLite C plus backup, chrono, and column metadata features.[^commands]
- Dataframes: Polars 0.55.2 belongs to separate `nu_plugin_polars`, not the parser/evaluator baseline; its manifest includes cloud providers, AWS credential machinery, and Tokio.[^polars]
- YAML: `nu-heavy-utils/yaml` enables `serde-saphyr`; the helper crate itself depends on Nu protocol types.[^heavy]
- Native SQLite C allocations are not automatically covered by Smith's Rust global allocator; accounting requires verification.

## Candidate inventory

Candidate verdicts are first-pass judgments, not approved interfaces.
Unmeasured costs and missing consumers remain open for every new facility.

| Feature family | Underlying libraries | Sample graph position | Independent SDK opportunity / review |
| --- | --- | --- | --- |
| File watch | `notify-debouncer-full`, `notify` | Optional `os` command feature | Strong candidate: granted roots, pull batches, overflow/rescan semantics, recorded event ordering, cancellation. |
| SQLite | `rusqlite`, `libsqlite3-sys` | Optional `sqlite` feature; bundled native C | Candidate plugin-local storage/query, not replacement session format; SQL authorization, transactions, interruption, row/byte limits, native allocation accounting. |
| Dataframes | Polars family, `sqlparser` | Separate Nu plugin | Conditional candidate: bounded columnar batches and query resources; isolate compute from filesystem/cloud access; investigate minimal feature set before adoption. |
| Numeric arrays | `ndarray` | Direct command dependency | Conditional numeric batch operations; establish consumers, shape semantics, allocation bounds, and boundary-copy costs. |
| Delimited records | `csv` | Direct command dependency | Candidate bounded streaming record parser; parsing does not imply filesystem access. |
| Spreadsheets | `calamine` | Direct command dependency | Conditional sheet readers; expansion limits, type/date mapping, complete versus streaming access. |
| XML | `roxmltree`, `quick-xml` | Direct command dependencies | Conditional tree/query or streaming parser; entity policy, depth bounds, one selected owner per concern. |
| JSON / Nu formats | `serde_json`, `nu-json`, `nuon` | Command dependencies | Reuse Smith JSON stack; Nu-specific protocol coupling is not an independent SDK contract. |
| YAML | `serde-saphyr` | Heavy-utils YAML feature | Conditional parser; explicit tags, aliases, scalar resolution, limits; do not expose Nu helper types. |
| TOML / KDL | `toml`, `toml_edit`, `kdl` | Direct command dependencies | Conditional parse/edit operations; existing TOML lock presence is not an application facade. |
| MessagePack | `rmp` | Direct command dependency | Conditional codec; do not replace or duplicate Smith's typed-session/CBOR contract. |
| HTML / JSON query / XPath | `scraper`, `gjson`, `sxd-document`, `sxd-xpath`, `webpage` | Separate query plugin | Conditional extraction/query; pure parsing separate from network grants; avoid another HTTP stack. |
| Mail / calendar / INI / plist | `eml-parser`, `ical`, `rust-ini`, `plist` | Separate formats plugin | Format-specific candidates only with actual consumers; not evaluator-inherent dependencies. |
| Markdown query | `mq-markdown` | Direct command dependency | Potential docs extraction/query; inspect behavior, limits, and existing documentation needs before adding an engine. |
| Search / fuzzy / regex | `fff-search`, `nucleo-matcher`, `fancy-regex` | Commands/utils; `fff-search` under `os` | Reuse existing `ignore` search owner first; distinct fuzzy/regex operations need explicit semantics and bounded work. |
| Compression | `brotli`; transport gzip through `ureq` features | Commands / optional network feature | Conditional bounded byte-stream codec; expansion limits, cancellation, no extra transport owner. |
| Hashes / byte encodings | `digest`, `sha2`, `md-5`, `base64`, `data-encoding` | Direct command dependencies | Candidate pure bounded transforms; graph versions differ; MD5 is not a security-integrity primitive. |
| Text encodings | `encoding_rs`, `chardetng` | Direct command dependencies | Candidate explicit decode/detect operations; detection uncertainty and invalid-byte policy must remain visible. |
| Dates / time zones | `chrono`, `chrono-tz`, `dtparse`, `human-date-parser` | Direct command dependencies | Conditional pure parse/format; pinned locale/timezone semantics; no ambient current time. |
| Glob / path / timestamps | `nu-glob`, `wax`, `filetime`, `uucore`, `uu_*` | Commands/system; several `os` gates | Existing file/search mechanisms first; investigate only gaps, symlink scopes, matching semantics, mutation evidence. |
| Host / process telemetry | `sysinfo`, `procfs`, platform helpers | Commands/system, target-specific | Conditional narrow telemetry; sensitive data grants, platform vocabulary, recorded nondeterministic input. |
| HTTP / TLS / DNS | `ureq`, `rustls`, `dns-lookup` | Optional network/TLS features | Reuse existing `smith-harness` HTTP engine; independent DNS only for demonstrated need. |
| Git metadata | `git2` | Separate `nu_plugin_gstat` | Conditional repository inspection; native build and repository scope review; no automatic second VCS implementation. |
| Clipboard / trash / launch | `arboard`, `trash`, `open` | Target-specific or optional OS features | Product-specific granted effects, not automatic SDK inventory; privacy, destructive behavior, supervision. |
| LSP / DAP / MCP | `lsp-server`, `lsp-types`, `rmcp`, `schemars`; Nu adapters | Optional top-level tooling components | Protocol/library reuse only when frontend requirements justify it; Nu adapters are not generic SDK APIs; MCP adds async/HTTP machinery. |
| Terminal/editor support | `ratatui`, `crossterm`, `reedline`, `edtui`, table/color helpers | CLI/pager components | Reuse existing terminal/UI vocabulary; do not expose engine widgets or pull an interactive Nu shell into the SDK. |
| Internal/support helpers | Nu protocol/parser utilities, macro, string, locale, sorting, MIME, endian/merge helpers | Mixed command/utility dependencies | Review individual pure primitives if needed; no one-to-one SDK API per dependency, no automatic public Nu helper layer. |

### Existing lock overlaps

- HTTP: `ureq` 3.4.2, `rustls` 0.23.45; existing engine owner.
- JSON/UI: `serde_json` 1.0.151, `ratatui` 0.30.2, `crossterm` 0.29.0; existing sanctioned concerns.
- TOML: `toml` 0.8.23 through `cargo-util-schemas`, `toml_edit` 0.22.27 through TOML.
- Regex: `fancy-regex` 0.11.0 through `termwiz`; Nu sample requests 0.19.
- Hashes: `sha2` 0.10.9 and `digest` 0.10.7 through Wasmtime/terminal dependencies; Nu sample requests 0.11 lines.
- Encoding: `encoding_rs` 0.8.42 through Wasmtime.
- Base64: 0.22.1 and 0.23.1 through terminal/HTTP dependencies; reconcile exact features and versions before claiming shared code.

Presence alone proves neither zero incremental cost nor a reusable public API.

## Target relationship

Independent SDK access reuses a library, not Nushell evaluation.

```mermaid
flowchart TB
  sdk["Plugin SDK"] --> facade["Host facade"]
  facade --> library["Shared crate"]
  nu["Nu"] --> library
```

The actual selected feature graph determines whether the crate is shared or newly introduced.

## Reproduction / continuation

1. Resolve the selected Nushell embedding crates/features, separately from CLI and optional plugin graphs.
2. Map each reachable feature to its underlying library and actual command callsites; classify optional additions explicitly.
3. Compare resolved target/runtime versions and features with Smith ownership; exclude dev/build-only overlap.
4. For every candidate, identify two real consumers and independently useful Smith operations, or record rejection/defer rationale.
5. Inspect license/advisory/native-build/musl support, grants, determinism, recording, replay, cancellation, and bounds.
6. Measure incremental build size/time, startup, resident memory, allocations, and native-versus-Nu/WIT invocation costs under matched conditions before claiming a benefit.
7. Promote only proven needs into spec work; no automatic public API or dependency adoption.

Any adopted library becomes an explicit sanctioned workspace dependency behind one owner/facade, with Smith types at boundaries.
Existing HTTP, search, JSON, session codec, and UI owners remain authoritative.
No new application Cargo features or interpreter dependency for independent SDK consumers.

## Evidence limitations

This covers feature families from sampled command/utility and named plugin/tooling manifests, not every transitive crate or Nushell command.
Implementation-to-library mapping was verified for watch, SQLite, selected format readers, and hashes; other rows retain manifest-level evidence.
Full target-specific resolution, license review of every candidate, real consumers, native prototypes, musl builds, allocation accounting, and measurements remain outstanding.
No specs, code, dependencies, or budget pins changed by this investigation.

## Completion criteria

- [ ] Selected embedding graph and optional additions have a complete classified inventory.
- [ ] Every candidate has verified callsites, graph membership, consumers, compatibility/cost evidence, and an adopt/reject/defer rationale.
- [ ] Proposed SDK facilities are independent of Nu evaluation and respect existing effect and world-shape contracts.

## Related contracts

- [Smith Specification](../../specs/SMH-SPEC-SPEC0001-smith-specification/index.md): reusable native SDK facilities, plugin-owned tools, durable effects.
- [Embedded Nushell](../../specs/SMH-SPEC-ZVMD_XYT-embedded-nushell/index.md): evaluator parity and effect safety; no claim that optional plugins belong to the evaluator.
- [Plugin System](../../specs/SMH-SPEC-PLUG0001-plugin-system/index.md): granted interfaces, Smith vocabulary, pull/batch semantics, two real consumers.

[^commands]: [Nu command dependencies/features](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-command/Cargo.toml), [workspace versions](https://github.com/nushell/nushell/blob/0.116.0/Cargo.toml), [watch implementation](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-command/src/filesystem/watch.rs), [SQLite implementation](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-command/src/database/values/sqlite.rs).
[^polars]: [Dataframe plugin dependencies/features](https://github.com/nushell/nushell/blob/0.116.0/crates/nu_plugin_polars/Cargo.toml), [plugin registration](https://github.com/nushell/nushell/blob/0.116.0/crates/nu_plugin_polars/src/lib.rs).
[^heavy]: [Heavy-utils YAML dependency/features](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-heavy-utils/Cargo.toml).

Selected command evidence: [CSV reader](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-command/src/formats/from/csv.rs), [XML reader](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-command/src/formats/from/xml.rs), [SHA-256](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-command/src/hash/sha256.rs).

Additional manifest evidence: [extra commands](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-cmd-extra/Cargo.toml), [utilities](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-utils/Cargo.toml), [formats plugin](https://github.com/nushell/nushell/blob/0.116.0/crates/nu_plugin_formats/Cargo.toml), [query plugin](https://github.com/nushell/nushell/blob/0.116.0/crates/nu_plugin_query/Cargo.toml), [Git plugin](https://github.com/nushell/nushell/blob/0.116.0/crates/nu_plugin_gstat/Cargo.toml), [system support](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-system/Cargo.toml), [LSP](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-lsp/Cargo.toml), [DAP](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-dap/Cargo.toml), [MCP](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-mcp/Cargo.toml), [pager/editor](https://github.com/nushell/nushell/blob/0.116.0/crates/nu-explore/Cargo.toml).