feat(array): builders no longer canonicalize their children [builders-child-stack]

[robert3005]

Rationale for this change

Canonical form is not recursive — Canonical only promises a canonical top level. Nested builders did not honour that: every appended child array was pushed through append_to_builder, which decoded it into the child's canonical builder. Struct fields, list elements and extension storage were all decompressed on the way through a builder that had no reason to look at them.

This is the first PR in a stack. The eventual goal is to delete the hand-written pack_struct_chunks / swizzle_list_chunks / swizzle_fixed_size_list_chunks special cases in ChunkedArray's canonicalization and route every dtype through append_to_builder — those helpers exist only because the builders used to decode children, and their doc comments describe exactly what this PR makes the builders do.

What changes are included in this PR?

ChildBuilder (vortex-array/src/builders/child.rs) accumulates a nested builder's child from the two sources such a builder receives:

• appended arrays are kept as chunks, in whatever encoding they arrived in
• appended scalars (and zeros/nulls/defaults) are materialized into a canonical builder

finish stitches the two back together, returning a ChunkedArray only when more than one chunk accumulated. StructBuilder, ListBuilder, ListViewBuilder, FixedSizeListBuilder and ExtensionBuilder now hold their children this way. Those are all the builders with array children; bool/primitive/decimal/varbinview/null have none, and Union/Variant builders do not exist yet.

Arrays shorter than MIN_CHUNK_LEN (64) are still materialized. Nested builders routinely append the elements of a single list, and a chunk per list would produce a ChunkedArray with more chunks than values.

ChildBuilder::set_validity_unchecked pushes the override into each chunk as a MaskedArray rather than decoding. That path panics if a chunk already contains nulls, since replacing its validity would mean decoding it; it is documented on the method. ExtensionBuilder is the only builder that forwards set_validity into a child — the others keep their own validity buffer — and the fast path (nothing chunked yet) is unchanged.

Tests

22 new tests. Beyond "children survive undecoded" for each nested builder, they cover the arithmetic and validity handling that the chunk path changes:

• empty arrays never become chunks; dtype mismatches are rejected on both sides of the length threshold; scalars, zeros and nulls appended around a chunk keep their order
• set_validity_unchecked slices the override per chunk, covers values still pending in the scalar builder, is dropped for a non-nullable child, and panics on a chunk that already contains nulls
• list offsets are rebased onto the running element count once elements live in chunks — for ListViewArray and ListArray inputs and for ListBuilder::append_array_as_list
• nested builders keep their own validity alongside a chunked child, extension storage receives a validity override through its chunks, and a built array with chunked children still canonicalizes recursively

Each test was checked against a mutated implementation. Dropping the per-chunk validity slice, dropping the pending flush, zeroing the listview offset base, and taking the ListBuilder offset from the appended array instead of the running total each fail the intended tests. Reverting to the old always-materialize behaviour fails 17 of them.

What APIs are changed? Are there any user-facing changes?

No public API changes — ChildBuilder is pub(crate). The user-facing change is the shape of what builders return: ArrayBuilder::finish is now documented as canonical at the top level only, and callers that need a fully decoded tree should ask for one via RecursiveCanonical (arrow export and the DuckDB exporter already recurse per child, so both are unaffected).

Checks

• cargo test -p vortex-array — 3164 lib + 72 doc tests
• cargo test --lib --tests green for vortex-arrow, vortex-row, vortex-btrblocks, vortex-compressor, vortex-layout, vortex-scan, vortex-file, vortex-ipc, vortex, vortex-datafusion, vortex-json, vortex-tensor, and all encodings
• cargo check --all-targets green for vortex-ffi, vortex-jni, vortex-tui, vortex-python-abi, vortex-fuzz
• cargo +nightly fmt --all, cargo clippy -p vortex-array --all-targets --all-features
• Stress run with MIN_CHUNK_LEN forced to 1 (every append becomes a chunk): 3141/3142 lib tests pass. The one failure, uncompressed_size_in_bytes::list_matches_materialized_size, compares against builder.finish().nbytes() — it measures the builder's compaction, which is what this PR removes. It passes at the real threshold.

Not run in this environment: vortex-duckdb and vortex-sqllogictest (DuckDB source download is blocked), CUDA crates, vortex-bench (lance-encoding needs protoc), and the Python suite.

---

Generated by Claude Code

Stacked PR Chain: builders-child-stack

PR	Title	Merges Into
#8964	feat(array): builders no longer canonicalize their children [builders-child-stack]	N/A
#8965	feat(array): let callers choose a nested builder's chunk threshold [builders-child-stack]	#8964
#8966	perf(array): accumulate nested builder validity without a null buffer [builders-child-stack]	#8965
#8967	refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack]	#8966

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will regress 1 benchmark

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 7 improved benchmarks
❌ 1 regressed benchmark
✅ 1841 untouched benchmarks
⏩ 46 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	canonicalize_sparse_fixed_size_list[(1024, 17, 8)]	327.3 µs	363.9 µs	-10.06%
⚡	Simulation	extend_from_array_zctl[(1000, 64)]	1,193.7 µs	518.1 µs	×2.3
⚡	Simulation	extend_from_array_zctl[(10000, 8)]	2.2 ms	1.4 ms	+58.97%
⚡	Simulation	extend_from_array_zctl[(1000, 8)]	401.9 µs	312.6 µs	+28.55%
⚡	Simulation	decode_varbin[(1000, 8)]	34.1 µs	27 µs	+26.12%
⚡	Simulation	extend_from_array_non_zctl_overlapping[(1000, 32)]	893.7 µs	719.2 µs	+24.26%
⚡	Simulation	extend_from_array_non_zctl_overlapping[(10000, 8)]	2.5 ms	2 ms	+23.83%
⚡	Simulation	extend_from_array_non_zctl_overlapping[(1000, 8)]	437.8 µs	379 µs	+15.5%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/builders-canonical-children-9ze0t6 (b4cb8fb) with develop (a12c310)}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[robert3005]

I am not sure this is real, need to see which benchmarks regress