perf(array): accumulate nested builder validity without a null buffer [builders-child-stack]

[robert3005]

Rationale for this change

Stacked on #8964 — review that first.

A nested builder learns about validity from two sources: one row at a time as scalars are appended, and a whole array's worth at a time as arrays are. Only the first needs a null buffer. LazyBitBufferBuilder treated both the same, so every appended array paid

self.nulls.append_validity_mask(&array.validity()?.execute_mask(array.len(), ctx)?);

— executing the array's validity into a Mask and copying its bits into a running buffer.

StructArray::try_concat has never done that. It hands its per-chunk validities to Validity::concat, which returns AllValid/AllInvalid/NonNullable when they are uniform and a Validity::Array(ChunkedArray<Bool>) otherwise — no bitmap, no execution. That gap is the one thing standing between the builders and the swizzle helpers they are meant to replace: routing chunked struct canonicalization through StructBuilder today would trade a lazy validity for a materialized one on every nullable chunked struct, which is the kind of regression that shows up in a scan rather than in a test.

What changes are included in this PR?

ValidityBuilder (vortex-array/src/builders/validity.rs) mirrors ChildBuilder: appended validity is kept as a run, bits appended one at a time go into a LazyBitBufferBuilder, and finish flushes the buffer into the run list and concatenates. StructBuilder, ListBuilder, ListViewBuilder and FixedSizeListBuilder hold their validity this way; the leaf builders keep LazyBitBufferBuilder unchanged.

The call sites lose their execute_mask:

self.nulls.append_validity(array.validity()?, array.len());

append_validity needs neither the ExecutionCtx nor a VortexResult any more, since nothing is executed on this path.

Two details worth calling out for review:

• However few values a run covers, it is kept as it arrived, so a builder's validity splits on exactly the boundaries its children do (feat(array): builders no longer canonicalize their children [builders-child-stack] #8964). Uniform runs still collapse, so a builder fed an array at a time does not pay a bool array for validity it never had.
• Validity::concat treats NonNullable and AllValid as different kinds and falls back to a bool array when both appear — which happens as soon as a non-nullable array is appended next to a scalar. Both mean "no nulls", so finish_with_nullability checks definitely_no_nulls across the runs first and answers from the declared nullability instead. Without that, appending an array and then a scalar to a non-nullable builder produced a Validity::Array and tripped the nullability assertion.

Seven unit tests on ValidityBuilder (runs preserved, uniform runs collapsing, a one-row validity earning a run too, bits and runs interleaving in order, set_validity replacing runs, non-nullable finishing non-nullable, finish resetting), plus a builder-level test asserting that two appended nullable struct arrays come back with a chunked bool validity rather than one copied buffer.

What APIs are changed? Are there any user-facing changes?

None — ValidityBuilder is pub(crate). The observable change is representational: a nested builder that consumed nullable arrays now returns Validity::Array(ChunkedArray<Bool>) where it previously returned a single BoolArray. That is the same shape StructArray::try_concat has always produced.

Checks

• cargo nextest run --workspace — 7168 passed, 560 skipped
• cargo clippy --all-targets --all-features — clean
• cargo +nightly fmt --all --check — clean

---

Generated by Claude Code

Stacked PR Chain: builders-child-stack

PR	Title	Merges Into
#9046	perf(array): stop flattening appended list views in ListViewBuilder [builders-child-stack]	N/A
#8964	feat(array): builders no longer canonicalize their children [builders-child-stack]	#9046
#8965	feat(array): let callers choose a nested builder's chunk threshold [builders-child-stack]	#8964
#8966	perf(array): accumulate nested builder validity without a null buffer [builders-child-stack]	#8965
#8967	refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack]	#8966

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will degrade performance by 32.39%

❌ 3 regressed benchmarks
✅ 1838 untouched benchmarks
⏩ 55 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	canonicalize_sparse_list[(512, 7, 4)]	2.2 ms	3.7 ms	-40.84%
❌	Simulation	canonicalize_sparse_list[(1024, 17, 8)]	1.9 ms	3 ms	-38.54%
❌	Simulation	canonicalize_sparse_fixed_size_list[(512, 7, 4)]	606 µs	712.9 µs	-14.99%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/builders-lazy-validity-9ze0t6 (a8ebf30) with claude/builders-canonical-children-9ze0t6 (e45996f)²}

Footnotes

1. 55 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on claude/builders-canonical-children-9ze0t6 (df75205) during the generation of this report, so 0cc201e was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩