Playground: re-provision systems whose scripts changed in the repo

[puzpuzpuz]

The symptom

Two of the playground's own QuestDB examples fail:

$ curl -sS -X POST 'https://clickbench-playground.clickhouse.com/api/query?system=questdb' \
       --data-binary "SELECT length_bytes('abc');"
{"error":"unknown function name: length_bytes(STRING)","position":7}

length_bytes() is what Q28 and Q29 in questdb/queries.sql use, and the
playground offers both from its Example dropdown.

Why

The engine there is not the one the repo describes:

$ ... --data-binary 'SELECT build();'
Build Information: QuestDB 9.3.1, JDK 17.0.9, Commit Hash 5480b2d...

• questdb/install has pinned 9.3.5 since Update QuestDB to 9.3.5 #902 (merged 2026-06-30).
• questdb/queries.sql switched Q28/Q29 to length_bytes() in 5305bf2
  (2026-03-05), to match ClickHouse's byte semantics — QuestDB's length()
  counts characters on VARCHAR. length_bytes() first shipped in
  9.3.2.
• The playground snapshot is from ~2026-05-12, when the repo still pinned
  9.3.1.

So the VM is two months behind the scripts. The mismatch is structural
rather than QuestDB-specific: handle_queries reads queries.sql from the
live repo checkout on the host, while the engine answering comes from a
snapshot built whenever that system was first provisioned. Nothing
reconciles the two — provision-all.sh skips every system that already has
a snapshot, and INSTALL.md only documents re-provisioning after agent
or base-image changes. A system's own install/create.sql/queries.sql
can move arbitrarily far without anything noticing.

The fix

• systems.py: source_fingerprint() — SHA-256 over the files that
  actually reach the VM. Excludes results/, template.json and READMEs,
  mirroring the --exclude list in build-system-rootfs.sh; otherwise every
  auto-results commit would mark the catalog stale.
• vm_manager.py: stamp it at <state>/systems/<name>/source.sha256 after
  a successful provision, and add VMManager.source_stale().
• main.py: expose source_stale on /api/system/<name>. Deliberately not
  on /api/state — that one is polled every 2 s by every open tab, and this
  hashes files.
• provision-all.sh: re-kick stale systems even under SKIP_PROVISIONED=yes,
  so git pull && provision-all.sh picks up engine bumps.
• INSTALL.md: document it.

Existing snapshots have no stamp, so they fall back to comparing the
system's newest source mtime against snapshot.state's. That keeps adopting
this from re-provisioning all 120 systems at once — only the ones whose
scripts really moved get re-kicked.

Also: questdb/install began with [ -d questdb/bin ] && exit 0, which made
it a permanent no-op — any working directory that survives a run keeps the
first version installed no matter how QDB_VERSION moves. Now it stamps the
installed version and reinstalls when it differs (clearing the old tarball
too, since the untar globs are questdb*.tar.gz).

Testing

• source_stale() against a temp state dir over six cases: no snapshot,
  legacy snapshot with newer/older scripts, stamped and unchanged, stamped
  with install edited, and stamped with a new results/ file added (must
  stay fresh). All as expected.
• provision-all.sh's skip decision for snapshotted/ready × fresh/stale,
  down, and a server that doesn't yet report the field (skips, as before).
• questdb/install with stubbed wget/tar: fresh install stamps 9.3.5;
  re-run makes no network calls; QDB_VERSION=9.4.3 re-downloads, replaces
  the tarball and re-stamps; a questdb/bin with no stamp reinstalls.

Not exercised: an actual provision run (needs the Firecracker host).

After merging

The QuestDB snapshot won't fix itself — it needs one re-provision:

curl -X POST http://localhost:8000/api/admin/provision/questdb

Other systems bumped since their snapshot was taken will be picked up by the
next provision-all.sh run. Happy to split the questdb/install change into
its own PR if you'd rather keep this to the playground.

[puzpuzpuz]

Closing this — the immediate problem is an operational one (the QuestDB snapshot needs a re-provision), so I've moved it to #1146 rather than leading with a patch to your infrastructure.

The mechanism here (fingerprint each system's scripts at provision time, expose source_stale on /api/system/<name>, let provision-all.sh re-kick stale systems) is written and tested if you decide you want a staleness check. Happy to reopen.