v0.1.10-4: Add Unified v1 LMCache Gateway, Tier, Adapter, and Queue Observability

[rickisba]

Background

The v1 architecture keeps the high-frequency KV data path between vLLM, LMCacheMPConnector, and LMCache MP. KDN observes and controls LMCache through a versioned Gateway rather than inferring cache state from Redis keys or reimplementing storage maintenance.

Later Proxy queueing, knowledge-aware policy, and network-compute overlap require a trace that separates:

• KDN knowledge resolution and policy;
• KDN Cache Service Facade operations;
• LMCache Gateway capability discovery and requests;
• LMCache L1/L2, adapter, tier, prefetch, pin, clear, and operation observations;
• Instance-side actual hit tokens and remote reads;
• Proxy queueing and vLLM execution;
• Legacy compatibility projections.

Objective

Introduce unified request and cache-operation traces for the v1 LMCache-backed path while preserving the existing Legacy path and current routing/forwarding behavior.

Required Stages

When applicable, the schema should represent:

• Runtime Profile resolution and freeze;
• KDN knowledge lookup and semantic resolution;
• Artifact compatibility evaluation;
• LMCache Endpoint and Capability Snapshot discovery;
• token/artifact lookup request and result;
• cache-object/tier/adapter observation;
• logical Prefetch/Pin/Unpin/Clear/Rebuild task queue and execution;
• LMCache MP HTTP/Coordinator/SDK request timing;
• LMCache asynchronous operation status/completion;
• L1/L2 or adapter-specific observations where publicly available;
• network/remote-read measurements reported by LMCache;
• Instance-side LMCache load and actual hit-token confirmation;
• CachePlan/FusionPlan construction;
• Prefill queue and execution;
• Decode execution/completion;
• fallback selection and execution;
• Legacy scan/dump/restore/inject stages, explicitly marked as Legacy.

Provenance Fields

Add explicit fields for:

• resolved runtime_profile;
• lmcache_profile_id;
• lmcache_endpoint_id and endpoint generation;
• Gateway adapter/profile;
• storage adapter and tier where available;
• predicted_*, observed_*, actual_*, and inferred values;
• source and source endpoint;
• observation timestamp, freshness/age, and expiry;
• request start/end/duration;
• queue and execution duration;
• token lookup range and matched coverage where available;
• actual hit tokens and remote-read counters where available;
• stage outcome;
• unsupported/incompatible/stale/partial status;
• error and fallback reason;
• Legacy projection marker.

Implementation Scope

• Define stable JSON-serializable request and CacheOperationTask trace models.
• Link reusable cache-operation tasks to all waiting requests through IDs.
• Distinguish KDN decisions from Gateway invocation, LMCache runtime observation, Proxy inference, and vLLM measurement.
• Distinguish L1/L2/tier/adapter observations without copying LMCache's physical chunk index.
• Prefer LMCache-native metrics, events, status, and hit-token observations over latency inference or Redis-key existence.
• Use monotonic clocks for durations and wall-clock timestamps only for external correlation.
• Preserve trace context across current Scheduler, Proxy, KDN, Instance, and Gateway boundaries.
• Allow tracing to be disabled or sampled without affecting correctness.
• Keep v1 fields optional for text-only and Legacy paths.
• New observability features target v1; Legacy receives only the fields needed to preserve regression visibility.

Acceptance Criteria

• A request exports a deterministic machine-readable stage timeline.
• A shared LMCache-backed operation can be correlated with all waiting requests.
• Runtime Profile is explicit and immutable for the trace lifetime.
• Predicted, observed, measured, inferred, stale, unsupported, partial, skipped, and Legacy-projected values cannot be confused.
• Every physical cache observation identifies source, profile, endpoint generation, and freshness where available.
• Token/artifact lookup timing is separate from prefetch/load timing and vLLM compute.
• LMCache Gateway request time is separate from LMCache asynchronous operation time.
• Actual cache reuse can be confirmed by hit-token or remote-read metrics where the active Profile exposes them.
• Missing, unsupported, incompatible, stale, failed, cancelled, and skipped stages are explicit.
• Stage durations are non-negative and use consistent units.
• Trace collection does not alter routing, cache reuse, injection, forwarding, or fallback decisions.
• A v1 trace cannot silently contain a Legacy write stage.
• Text-only, Legacy KV, v1 LMCache-backed KV, unknown-future Profile, fallback, and endpoint-restart paths are representable.
• Unit tests cover serialization, ordering, shared-task correlation, endpoint generation, stale observations, unsupported capability, failed/skipped stages, and provenance separation.
• At least one CPU-only integration test produces an end-to-end timeline using Mock Gateway/Profile data.

Non-Goals

• Implementing the production LMCache Gateway I/O.
• Implementing ExecutionGraph scheduling.
• Maintaining an authoritative Proxy or KDN physical block index.
• Reimplementing LMCache metrics, events, tier accounting, or eviction.
• Claiming queueing or cache-performance improvement in v0.1.10.
• Adding new Legacy functionality.
• Requiring an external tracing backend.

Parent Epic

• v0.1.10 Epic: KDN Contracts, LMCache Compatibility, and Observability Baseline #137

Dependencies

• v0.1.10-2: Define v1 Runtime, Artifact, LMCache Observation, Operation Task, and Queue State Models #139 for state/object naming.
• v0.1.10-3: Define Versioned KDN Cache Service Facade and LMCache Gateway Contracts #140 for Cache Service, Gateway, Profile, and capability vocabulary.