perf(array): stop flattening appended list views in ListViewBuilder [builders-child-stack]

[claude]

Rationale for this change

New bottom of the builders-child-stack. This is the change that makes the rest of the stack pay off, so it lands first.

ListViewBuilder::append_listview_array rebuilt every incoming ListViewArray into an exact layout before appending it:

let listview = array.into_owned().rebuild(ListViewRebuildMode::MakeExact, ctx)?;

Rebasing every offset by the number of elements already in the builder is correct whatever layout the source has, so that rebuild bought nothing but an unconditional promise that the finished array is zero-copyable to a ListArray. What it cost was any sharing the source had gone to the trouble of expressing, since MakeExact is documented as "removing all data overlaps and creating a flattened layout".

The case that matters is a constant list array. constant_canonical_list_array already does the clever thing — "canonicalize only applies to the top level array, so we can simply have 1 scalar in our child elements and have all list views point to that scalar" — and the builder immediately undid it. Measured on a 10,000-row fill of a 3-element list:

	elements materialized	nbytes
appending the same list in a loop	30,000	280 KB
the constant array, canonicalized alone	3	16 B
the constant array, appended (before)	30,000	400 KB
the constant array, appended (after)	3	160 KB

That unblocks callers replacing per-row append loops with a single array append — encodings/sparse's append_list_fill calls append_array_as_list(fill_elements, ctx) once per filled row, which for a sparse list array of a million rows is a million copies of the same handful of values.

What changes are included in this PR?

append_listview_array keeps trimming unreferenced elements — ListViewRebuildMode::TrimElements preserves overlap and the existing flag — but otherwise appends the views as they arrived.

ListViewBuilder gains a zero_copy_to_list field that starts true and is cleared when an appended ListViewArray brings a layout the builder chose not to rewrite. finish_into_listview reports it instead of hardcoding true. Every other append lays its lists down back to back, so only this one path can clear it.

The test is deliberately conservative: the result stays zero-copyable only if the incoming views are packed back to back and reference every element they carry, since the next append starts where this one's elements end. Leading or trailing unreferenced elements would leave a gap in the middle of the combined array.

What APIs are changed? Are there any user-facing changes?

No signature changes. Two behavioural ones, both intentional:

• A builder fed a non-exact ListViewArray now finishes non-zero-copy-to-list. The flag is per-array and consumers already branch on it (vortex-duckdb/src/exporter/list_view.rs:59 has both paths); callers who need an exact layout can rebuild. test_extend_from_array_overlapping_listview is updated to assert the new contract, including that a null list keeps its source size metadata rather than being rewritten to zero.
• The builder no longer compacts elements unreferenced between lists. TrimElements only drops leading and trailing ones. uncompressed_size_in_bytes::list_matches_materialized_size used a builder round-trip as its proxy for "materialized" and relied on that compaction, so it now compares against an explicitly MakeExact-rebuilt array; the comment explains why.

Checks

• cargo test -p vortex-array — 3138 lib tests, including a new one asserting a 10,000-row constant list append stores its value once
• cargo test --lib --tests green for vortex-arrow, vortex-layout, vortex-scan, vortex-file, vortex-datafusion, vortex-btrblocks, vortex-compressor, vortex-row, vortex, vortex-ipc, vortex-sparse, vortex-fastlanes
• cargo +nightly fmt --all, cargo clippy -p vortex-array --all-targets --all-features, typos

---

Generated by Claude Code

---

Stacked PR Chain: builders-child-stack

PR	Title	Merges Into
#9046	perf(array): stop flattening appended list views in ListViewBuilder [builders-child-stack]	N/A
#8964	feat(array): builders no longer canonicalize their children [builders-child-stack]	#9046
#8965	feat(array): let callers choose a nested builder's chunk threshold [builders-child-stack]	#8964
#8966	perf(array): accumulate nested builder validity without a null buffer [builders-child-stack]	#8965
#8967	refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack]	#8966

[codspeed-hq]

Merging this PR will improve performance by 95.8%

⚡ 6 improved benchmarks
✅ 1835 untouched benchmarks
⏩ 55 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	extend_from_array_non_zctl_overlapping[(10000, 8)]	2,450.1 µs	704.9 µs	×3.5
⚡	Simulation	extend_from_array_non_zctl_overlapping[(1000, 32)]	899.5 µs	290.1 µs	×3.1
⚡	Simulation	extend_from_array_zctl[(10000, 8)]	2.2 ms	1.2 ms	+83.29%
⚡	Simulation	extend_from_array_zctl[(1000, 64)]	1,189.9 µs	727.9 µs	+63.46%
⚡	Simulation	extend_from_array_non_zctl_overlapping[(1000, 8)]	434.5 µs	293.6 µs	+47.98%
⚡	Simulation	extend_from_array_zctl[(1000, 8)]	398.4 µs	337.8 µs	+17.93%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/listview-builder-keep-overlaps-9ze0t6 (0779ba5) with develop (1c92be6)}

Footnotes

1. 55 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩