Router: API key auth middleware, LRU cache, and v2 metrics (NET-371/373/375)

[mo4islona]

Summary

• NET-371: auth middleware on GET /network/:dataset/:start_block/worker — parses Authorization: Bearer sqd_data_<id>_<rnd>, validates via cache → Network API, returns 403 with CREDENTIALS_INVALID body + WWW-Authenticate: Bearer realm="sqd-archive". Gated by ENFORCE_V2_AUTH (default off): the flag changes only the action — parsing, cache, and Network API calls run either way so adoption metrics stay clean.
• NET-373: in-process LRU cache (moka + own Clock trait for deterministic time tests) with three explicit states UNDEFINED / EXISTS / DELETED, asymmetric TTLs (60s / 15s), expires_at clamp, and singleflight (DashMap of per-key tokio::sync::Mutex) so concurrent misses for the same key produce one upstream call.
• NET-375: Prometheus metrics (sqd_v2_auth_total, sqd_v2_auth_latency_seconds, sqd_v2_cache_{hit,miss}_total, sqd_v2_validate_call_total, sqd_v2_requests_by_key with bounded top-100 sketch + explicit remove_label_values on eviction so cardinality stays bounded) plus sqd_router_worker_urls_handed_total for the Worker:Router ratio canary. key_id is the only token-derived value in logs/labels; the full token is never logged.

Resilience details

• Network API client: 250ms timeout, half-open circuit breaker (50 consecutive errors → open 30s → exactly one probe admitted; probe failure re-opens for another full window).
• Fail-open on outage with cached DELETED still denying.
• Strict token parser (rejects extra _, non-alphanumeric chars per Tech Plan §2.1 alphabet).

Wiring

• cli.rs: --enforce-v2-auth / ENFORCE_V2_AUTH, --network-api-url / NETWORK_API_URL.
• server.rs: middleware mounted only on the worker route via .route(...).layer(from_fn(auth)); /ping, /network/:dataset/height, /metrics stay open.
• main.rs: NetworkApiClient::disabled() if NETWORK_API_URL unset (dev-friendly: every request fails open).

Test plan

• cargo +1.89 test -p router — 51 passed, 0 failed
  • Cache (9): three states, TTL defaults, expires_at clamp, already-expired downgrade, capacity eviction, UNDEFINED ≠ DELETED regression
  • Client (12): 200/404/500/timeout/connection-error, malformed-success-body, missing-user_id, breaker open / probe / reset / half-open admits exactly 1 / probe failure re-opens
  • Middleware (20): Bearer extraction, no Token: fallback, cheap-reject, enforce on/off, bad-key flood (1 call/15s), timeout fail-open without cache write, cached Deleted denies during outage, breaker open passes through, key_id in extensions, constant-time mismatch, miss→hit, latency histogram, concurrent miss flood (32 racers → 1 API call), strict parser (extra _, missing rnd, non-alnum), malformed-body doesn't poison cache
  • Logging (2): full token never logged, warn carries key_id only
  • Top-keys (3): bounded cardinality, high-count keys retained, eviction signal
  • Singleflight (3): per-key serialisation, distinct keys parallel, no leak
  • Server (1): worker URL counter doesn't increment on 503
• Local smoke test against a wiremock standing in for Network API
• Staging soak: watch sqd_v2_validate_call_total{result="fail_open"} after deploy

🤖 Generated with Claude Code