Shared: Add YEAST desugaring library

[tausbn]

This PR adds a cleaned-up prototype of the YEAST library that was developed in a hackathon a few years ago.

YEAST is intended to be a lightweight layer for performing various kinds of AST cleanup and desugaring directly on the parse tree produced by a tree-sitter parser. Rewrite rules are specified declaratively, with a query language that approximates that of tree-sitter, though notably with no alternation or anchors (and also with greedy semantics -- no backtracking). I expect that this will be sufficient for most uses.

Output templates also look like tree-sitter trees, with embedded rust blocks for specifying code that calculates an AST based on the given input.

Because the output AST may be an entirely different language from the input AST, this PR also adds a new node-types.yml format -- a lightweight reformulation of node-types.json intended for human consumption (unlike the latter).

Of note: the output format disallows having field-less child nodes. The node-types.yml format supports them, but YEAST itself will silently throw them away.

---

There's a lot of code in this PR, but it's just a prototype, so don't feel compelled to review it in detail.

DO, however, look at the documentation, and also the changes to the existing tree-sitter extractor infrastructure (the final two commits).

[tausbn]

CI failure seems to be unrelated to my changes.

[Copilot]

Pull request overview

This PR introduces YEAST, a Rust library for declarative AST cleanup/desugaring on top of tree-sitter parse trees, and integrates it into the shared tree-sitter extractor so extraction can optionally run on a rewritten AST and/or validate against an alternate output node-types schema.

Changes:

• Add new shared/yeast + shared/yeast-macros crates implementing the rule/query/template system, plus tests and documentation.
• Extend the shared tree-sitter extractor to optionally run YEAST rules and to support separate output_node_types for schema generation and TRAP validation.
• Update Bazel vendored Rust deps to include YEAST dependencies (and bump tree-sitter, cc, etc.).

Show a summary per file

File	Description
shared/yeast/tests/test.rs	End-to-end tests for parsing, query matching, tree building, and desugaring rules.
shared/yeast/tests/node-types.yml	Test output schema in the new YAML node-types format.
shared/yeast/src/visitor.rs	Converts a tree-sitter Tree into a YEAST Ast.
shared/yeast/src/tree_builder.rs	Fresh identifier generation support for templates/rules.
shared/yeast/src/schema.rs	Schema representation for kinds/fields (language-derived or YAML-derived).
shared/yeast/src/range.rs	Serde helpers for (de)serializing tree_sitter::Range.
shared/yeast/src/query.rs	Query AST and matching engine (captures, repetition, named/unnamed semantics).
shared/yeast/src/print.rs	Debug printer for walking a YEAST AST.
shared/yeast/src/node_types_yaml.rs	YAML ↔ JSON node-types conversion + schema construction from YAML.
shared/yeast/src/lib.rs	Core YEAST types (Ast, Node, Rule, Runner) and rewrite application logic.
shared/yeast/src/dump.rs	Human-readable AST dump utility used by tests.
shared/yeast/src/cursor.rs	Cursor trait abstraction used by traversal/extractor integration.
shared/yeast/src/captures.rs	Capture storage and utilities (single/repeated/optional).
shared/yeast/src/build.rs	BuildCtx used by tree!/trees! macros to build synthetic nodes.
shared/yeast/src/bin/node_types_yaml.rs	CLI tool to convert YAML node-types ↔ JSON node-types.
shared/yeast/src/bin/main.rs	Minimal YEAST CLI for parsing and printing.
shared/yeast/doc/yeast.md	Main YEAST documentation (architecture, query/template language, integration).
shared/yeast/doc/node-types-yaml.md	Specification for the YAML node-types format and CLI usage.
shared/yeast/Cargo.toml	New yeast crate manifest and dependencies.
shared/yeast/Cargo.lock	Lockfile for the standalone shared/yeast crate.
shared/yeast/BUILD.bazel	Bazel target for the yeast Rust library.
shared/yeast/.gitkeep	Placeholder file for directory tracking.
shared/yeast/.gitignore	Ignores shared/yeast/target.
shared/yeast/.envrc	Direnv config for local development.
shared/yeast-macros/src/parse.rs	Proc-macro parsing and codegen for query!, tree!, trees!, rule!.
shared/yeast-macros/src/lib.rs	Proc-macro entry points and user-facing macro docs.
shared/yeast-macros/Cargo.toml	New yeast-macros proc-macro crate manifest.
shared/yeast-macros/BUILD.bazel	Bazel target for the yeast-macros proc-macro crate.
shared/tree-sitter-extractor/tests/multiple_languages.rs	Updates tests to include output_node_types in LanguageSpec.
shared/tree-sitter-extractor/tests/integration_test.rs	Updates tests to include output_node_types in LanguageSpec.
shared/tree-sitter-extractor/src/generator/mod.rs	Generator uses output_node_types when provided.
shared/tree-sitter-extractor/src/generator/language.rs	Adds output_node_types to generator Language.
shared/tree-sitter-extractor/src/extractor/simple.rs	Uses output_node_types for schema validation in the simple extractor.
shared/tree-sitter-extractor/src/extractor/mod.rs	Adds optional YEAST desugaring path and AstNode abstraction.
shared/tree-sitter-extractor/Cargo.toml	Adds a path dependency on shared/yeast.
shared/tree-sitter-extractor/BUILD.bazel	Adds Bazel dep on //shared/yeast.
ruby/extractor/src/generator.rs	Populates output_node_types: None for Ruby/Erb generator languages.
ruby/extractor/src/extractor.rs	Updates shared extractor invocation with new extract(...) params.
ql/extractor/src/generator.rs	Populates output_node_types: None for QL generator languages.
ql/extractor/src/extractor.rs	Populates output_node_types: None for QL simple extractor languages.
MODULE.bazel	Adds/upgrades vendored crates (notably tree-sitter and new deps).
misc/bazel/3rdparty/tree_sitter_extractors_deps/defs.bzl	Adds YEAST + YEAST-macros crates and bumps vendored dependencies.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.zstd-sys-2.0.16+zstd.1.5.7.bazel	Updates cc dependency reference.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.tree-sitter-ruby-0.23.1.bazel	Updates cc dependency reference.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.tree-sitter-ql-0.23.1.bazel	Updates cc dependency reference.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.tree-sitter-python-0.23.6.bazel	Adds vendoring/build definitions for tree-sitter-python.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.tree-sitter-json-0.24.8.bazel	Updates cc dependency reference.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.tree-sitter-embedded-template-0.25.0.bazel	Updates cc dependency reference.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.tree-sitter-0.26.8.bazel	Bumps vendored tree-sitter to 0.26.8 and updates cc reference.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.iana-time-zone-haiku-0.1.2.bazel	Updates cc dependency reference.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.find-msvc-tools-0.1.9.bazel	Updates vendored find-msvc-tools version metadata.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.cc-1.2.61.bazel	Updates vendored cc version metadata and dependencies.
misc/bazel/3rdparty/tree_sitter_extractors_deps/BUILD.bazel	Adds aliases for serde_yaml and tree-sitter-python; bumps tree-sitter alias.
Cargo.toml	Adds shared/yeast and shared/yeast-macros to the workspace.
Cargo.lock	Workspace lock updates (adds yeast crates; bumps tree-sitter, cc, etc.).

Copilot's findings

Comments suppressed due to low confidence (1)

shared/yeast/src/node_types_yaml.rs:303

• schema_from_yaml_with_language also registers YAML unnamed: tokens using schema.register_kind(name), which only affects the named kind map. If the YAML adds any unnamed tokens not present in the tree-sitter language, QueryNode::UnnamedNode lookups will still fail because unnamed_kind_ids is never updated.

This should use an unnamed-kind registration path (updating unnamed_kind_ids) rather than register_kind.

• Files reviewed: 52/55 changed files
• Comments generated: 6

[Copilot]

Copilot's findings

• Files reviewed: 51/54 changed files
• Comments generated: 3

[Copilot]

Copilot's findings

• Files reviewed: 49/50 changed files
• Comments generated: 2