Splat run-end bool decode into 64-bit words - #9021
Open
joseph-isaacs wants to merge 5 commits into
Open
Conversation
Merging this PR will regress 5 benchmarks
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
joseph-isaacs
marked this pull request as ready for review
July 27, 2026 18:13
joseph-isaacs
force-pushed
the
claude/runend-decompress-bools-1t2re2
branch
2 times, most recently
from
July 28, 2026 11:27
5edb616 to
13cedf9
Compare
The bool decoder prefilled the output with the majority value and then
`fill_range`d every run whose value differed. That pays, per run, a
data-dependent branch on the value plus a range fill that masks a partial
byte at each end -- work that is constant per run and so never amortized
when runs are short.
Add a splat kernel that accumulates runs into a 64-bit word and stores each
output word exactly once: the run's value only feeds an AND mask, so the
value branch disappears, and head/tail masking is paid per output word
instead of per run.
The two structures are complementary, so keep both and dispatch on the
number of runs the prefill would have to patch, which is what its skipping
buys it. Splat over prefill, by fastest sample on
`run_end_decode_bool_ablation` (65536 elements, randomised run lengths and
values, rotating datasets):
run length 1 2 8 32 64 128 1024
non-nullable
50/50 3.70 3.46 2.86 1.25 1.26 0.87 0.61
90/10 - 1.28 0.93 0.44 0.48 0.49 0.61
nullable
50/50 1.97 2.09 1.92 1.21 0.74 0.50 0.63
90/10 - 1.20 1.14 0.64 0.46 0.47 0.63
`prefer_splat` reproduces this table's sign at every point but non-nullable
8/90-10, where it gives up 7%.
The prefill kernel also now takes run ends as a slice and trims by offset
inline, rather than through `trimmed_ends_iter` plus a bit-at-a-time
`BitBuffer` iterator; per-run work here is a handful of instructions, so
that chain is not amortized by anything either.
Net against the previously shipped kernel, fastest sample: 3.9x at run
length 1, 3.0x at 8, 1.3-1.4x at 32-64, parity at 1024, and 2.0x for
nullable short runs.
Signed-off-by: Claude <noreply@anthropic.com>
The splat appended its output words to a `BufferMut<u64>` with `push` and
`push_n`. Size the buffer up front instead -- the final word count is known
exactly -- and have the loop overwrite it by index.
The two obvious reasons to expect a win are both wrong, and the disassembly
is what says so. `push_n` and `slice::fill` lower to the identical inlined
vector store loop; neither becomes a `memset`, because the splat value is a
runtime value rather than a byte pattern. And the per-word capacity check
does not disappear so much as trade places with a slice bounds check.
The win is second-order. `reserve`'s allocation slow path kept
`append_spanning` above LLVM's inline budget, so `&mut self` escaped into a
call and the accumulator lived on the stack: a read-modify-write of `word`
and a store-plus-reload of `bits` on every short run, on the loop's critical
dependency chain. Without that slow path the spanning path inlines and both
stay in registers. `#[inline]` is needed to get this in the crate build --
the bench mirror is a smaller function and inlines without it, which is
exactly the sort of gap a bench-only check would hide.
In-process, candidate over baseline by median sample:
run length 1 2 8 32 64 128 1024
non-nullable
50/50 0.94 0.94 0.87 0.79 0.68 0.74 0.84
90/10 - 0.94 0.87 0.78 0.63 0.71 0.80
nullable
50/50 0.96 0.97 0.91 0.81 0.78 0.59 0.82
90/10 - 0.98 0.94 0.79 0.79 0.53 0.88
Nullable 1- and 2-element runs are a wash rather than a win; everything else
is real, and one of the two nullable accumulators still spills under the
register pressure of driving two sinks.
Position is now advanced with `pos = end` rather than `pos = pos.max(end)`.
The guarded form put a `cmova` on the loop-carried position where a plain
`mov` breaks the chain, and at run length 1 that alone cost 15-25%. Bounds
-checked indexing keeps a malformed run-end array to a panic either way.
`prefer_splat`'s threshold was derived against the slower splat, so it is now
conservative in the sense that it hands the prefill grid points the splat
would win. Recorded in its doc comment; retuning it means re-deriving the
whole ablation table.
Adds `push_splat` to the ablation bench so the append-versus-overwrite
comparison stays reproducible.
Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
force-pushed
the
claude/runend-bool-decode-benches
branch
from
July 28, 2026 17:11
bd674b8 to
dbda9ee
Compare
The nullable splat drove two independent `BitSplat`s, one for the decoded
bits and one for validity. They are appended to in lockstep -- every run
contributes the same number of bits to both -- so all of the bookkeeping was
being done twice: two run lengths, two word-boundary tests, two head masks,
two spanning paths. The disassembly showed the doubled live state spilling as
well: ten memory operations per run against the non-nullable loop's zero.
Fuse them. One `bits` counter, one boundary test, one mask feeding both
accumulators. Measured in-process against the two-sink form, this is worth
0.86-0.70 on its own, and it is the largest single contributor on the
nullable side.
Also settles the inline attribute on the spanning paths. `#[inline]` is not
enough: the crate build leaves the function out of line even where an
identical bench-local copy is inlined, and out of line means `&mut self`
escapes and the accumulator lives on the stack for the whole loop. With
`#[inline(always)]` no call to either spanning path survives in the object
file, which is the property the comment now claims and `objdump` confirms.
Run position now advances by the clamped run length rather than jumping to
the run end. The sinks are sized to exactly `length` bits, so a position that
could move backwards would let the total appended exceed the buffer and turn
a malformed array into an index-out-of-bounds panic -- and run ends are only
checked for sortedness under `debug_assertions`, so a release build does see
them. `decode_bools_non_monotonic_ends` covers it.
With the splat 1.3-1.9x faster than when `prefer_splat`'s constant was fitted,
that constant is re-derived. The crossover moved out: the splat now wins at
8-element runs with 90/10 values, and at 32- and 64-element runs with even
values, all of which the old threshold handed to the prefill. A second output
buffer also no longer doubles the splat's cost, since the fused pair shares
the per-run bookkeeping, so the weight is `buffers + 1` rather than `buffers`.
The new constant picks the faster kernel at 25 of the 26 grid points, against
24 before; the miss is non-nullable 1024-element runs at 50/50, worth 6%.
Net against the kernel this PR started from, by fastest sample:
run length 1 2 8 32 64 128 1024
non-nullable
50/50 4.14 3.93 3.35 1.87 1.20 1.02 1.39
90/10 - 1.60 1.24 1.14 1.37 1.38 1.68
nullable
50/50 2.51 2.61 2.51 1.59 1.22 1.23 1.00
90/10 - 1.46 1.36 1.05 1.11 1.08 1.04
Signed-off-by: Claude <noreply@anthropic.com>
The splat tests, per run, whether the run reaches the end of the current
word. That test is data-dependent, and its predictability tracks the run
length: for runs averaging r bits it fires about r/64 of the time. Well below
a word it almost never fires and predicts perfectly. Around half a word it is
a coin flip, and the mispredictions dominate the loop -- which is why the
per-run cost peaked at run length 32 rather than falling monotonically.
Add a form with no such test. It paints the run from the current bit offset
all the way to the top of the word and stores unconditionally:
word = (word & low_mask(bits)) | (splat << bits);
words[idx] = word;
let completed = (bits + n) / 64;
...
idx += completed;
word = if completed > 0 { splat } else { word };
bits = (bits + n) % 64;
Bits above `bits` are don't-care, so painting past the run's end is free: the
next run blends over them. Two conditions make it correct. The output needs
one word of slack, because a zero-length run arriving once the buffer is
exactly full still stores -- and run ends clamped to the logical length
produce exactly that. And `finish` must mask the last partial word, or the
paint survives past the logical length where `true_count` would read it.
`blend_form_matches_naive` covers both, at three lengths chosen to land on
and off the word grid.
It is not strictly better, so both forms stay, picked by average run length,
which is O(1) from the run count. Blend over branching, by fastest sample:
run length 1 2 8 16 32 48 64 128 1024
non-nullable 0.75 0.79 1.02 1.35 1.71 1.39 1.15 0.86 0.74
nullable 0.82 0.88 1.15 1.37 1.71 1.79 1.74 1.89 1.16
Below a quarter of a word the redundant stores -- up to 64 per word of output
-- cost more than the branch. The nullable kernel crosses over earlier: it
does about twice the work per run, so the extra stores are proportionally
cheaper while a misprediction costs the same.
The blend path is `#[inline(never)]`. This is not a micro-preference: with
the two loops inlined into one function, the branching loop lost 20% with its
own text unchanged, measured against an in-process anchor that did not move.
Expressing the choice as a const generic over one loop did the same. Only
keeping the second loop out of line leaves the first one alone.
On the corpus benchmark, against the previous commit: run16 cases 1.16-1.37x,
run64 cases up to 1.50x, run4 cases unchanged.
Signed-off-by: Claude <noreply@anthropic.com>
The corpus and ablation generators now draw run lengths from a normal
distribution rather than a uniform band. The spread of run lengths, not the
mean, is what sets how often a run crosses a word boundary and how many runs
a prefill has to patch, so both dispatch constants were conditioned on the
old shape and had to be re-derived. Matching the ablation generator to the
corpus one also means the constants are fitted on the data CI measures.
The kernels separate far more cleanly on this data than on the uniform band.
Splat over prefill, by fastest sample:
run length 1 2 8 16 32 48 64 128 1024
non-nullable
50/50 3.57 3.67 3.71 3.75 3.45 3.02 2.91 1.52 0.85
90/10 - 1.38 1.35 - 1.14 - 0.90 0.74 0.76
Every win has at least 205 patched runs and every loss at most 102, so one
threshold separates them with margin, where the old fit had borderline points
either side. The nullable side splits at the same place: it doubles the work
on both sides of the comparison, since the prefill memsets and patches two
buffers exactly where the splat builds two. So `prefer_splat` loses its
`buffers` parameter entirely and becomes one patched run per eight output
words.
`prefer_blend` gains an upper bound. Blend over branching now reads
run length 1 2 8 16 32 48 64 128 1024
non-nullable 0.91 0.99 1.18 1.31 1.41 1.43 1.48 1.18 0.93
nullable 0.87 0.90 1.03 1.15 1.27 1.52 1.25 0.94 0.89
so the band has two edges, not one: far above a word the boundary test always
fires and predicts again, leaving the blend paying its extra store against a
whole-word fill that dominates either way. Without the bound the nullable
1024 cases came out 13% behind the original kernel, reproducibly across two
runs.
Against the original kernel, in-process, every point of the grid is now at or
above parity: non-nullable 1.01-4.05x, nullable 0.98-2.54x, the 0.98 being a
one-microsecond case. On the corpus benchmark, 1.14-3.87x across all 18.
Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
force-pushed
the
claude/runend-decompress-bools-1t2re2
branch
from
July 28, 2026 17:11
13cedf9 to
48ddd8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The bool decoder prefilled the output with the majority value and then
fill_ranged every run whose value differed. Per run that costs a data-dependent branch on the run's value — which mispredicts about half the time on 50/50 data — plus a range fill that masks a partial byte at each end around its memset. Both costs are constant per run, so when runs are short the loop isO(n · k)with nothing amortisingk.A run of bools is a bit pattern, so it can be built rather than patched: OR a mask of the run's value into a 64-bit accumulator and store each output word exactly once.
Result
On #9020's corpus benchmark, by fastest sample:
128x16k_run4_even128x16k_run4_even_nullable128x16k_run4_skewed128x16k_run16_even128x16k_run16_even_nullable128x16k_run16_skewed128x16k_run64_even128x16k_run64_even_nullable128x16k_run64_skewed4x100k_run4_even4x100k_run4_even_nullable4x100k_run4_skewed4x100k_run16_even4x100k_run16_even_nullable4x100k_run16_skewed4x100k_run64_even4x100k_run64_even_nullable4x100k_run64_skewedWhere the win comes from
Each commit measured on this same benchmark, at that commit. Every column is the speedup over the column to its left, so the five multiply to the total.
128x16k_run4_even128x16k_run4_even_nullable128x16k_run4_skewed128x16k_run16_even128x16k_run16_even_nullable128x16k_run16_skewed128x16k_run64_even128x16k_run64_even_nullable128x16k_run64_skewed4x100k_run4_even4x100k_run4_even_nullable4x100k_run4_skewed4x100k_run16_even4x100k_run16_even_nullable4x100k_run16_skewed4x100k_run64_even4x100k_run64_even_nullable4x100k_run64_skewedReading it honestly: the splat itself is most of the win, and the rest is worth having but is not where the headline comes from.
fuseis nullable-only by construction, so its non-nullable column is run-to-run noise around 1.0 and should be read as nothing rather than as a small loss.blendandrefithave to be read as a pair — the blend was landed with thresholds still fitted to the old uniform data, which cost up to 14% at 4-element runs, and the refit gives it back; together they are 0.94–1.36x, clearly positive at 16-element runs and neutral at 4.What changes are included in this PR?
1. The splat kernel — 1.03–3.71x, the bulk of the change. Accumulate into a
u64, flush whole words. Nothing is read back, and the run's value only feeds anANDmask, so the value branch is gone. Biggest where runs are short and values even, which is where the old kernel's per-run branch mispredicted most and its range fills were smallest; near nothing at 64-element nullable runs, where the prefill was already close to a memset.Both kernels are kept, because the prefill is not strictly worse: it skips majority runs entirely, so once patches are rare enough it approaches a bare memset and wins.
prefer_splatdispatches on the count its advantage depends on. The prefill also now takes run ends as a slice and trims by offset inline rather than throughtrimmed_ends_iterplus a bit-at-a-timeBitBufferiterator.2. Overwrite a pre-sized word buffer — 0.98–1.79x, broadly 1.03–1.28. Instead of appending with
BufferMut::push/push_n. The two obvious reasons to expect a win are both wrong, and the disassembly says so:push_nandslice::filllower to the identical inlined vector store loop (neither is amemset— the splat value is a runtime value, not a byte pattern), and the per-word capacity check trades places with a slice bounds check. The win is second-order: withoutreserve's allocation slow path, the spanning path fits LLVM's inline budget, so&mut selfstops escaping and the accumulator stays in registers.3. Fuse the nullable pair onto one bit counter — 1.23–2.00x on nullable cases, nothing elsewhere. The nullable path drove two independent sinks that are appended to in lockstep, so every run did the bookkeeping twice, and the doubled live state spilled: ten memory operations per run against the non-nullable loop's zero.
4. A branch-free splat form for word-scale runs — 0.86–1.19x on its own. The remaining per-run branch — does this run reach the end of the word — is data-dependent, and its predictability tracks the run length: for runs averaging
rbits it fires aboutr/64of the time. The new form paints the run from the current offset to the top of the word and stores unconditionally, relying on the bits above the offset being don't-care. It is not strictly better, so both forms stay, picked by average run length.5. Refit both constants for normally distributed run lengths — 0.96–1.19x, recovering commit 4's losses at short runs. Matches #9020's generator, and the ablation generator to it, so the constants are fitted on the data CI measures. The kernels separate far more cleanly on this data. Splat over prefill:
Every win has at least 205 patched runs and every loss at most 102, so one threshold separates them with margin where the previous fit had borderline points either side. The nullable side splits at the same place — it doubles the work on both sides of the comparison, since the prefill memsets and patches two buffers exactly where the splat builds two — so
prefer_splatloses itsbuffersparameter and becomes one patched run per eight output words.prefer_blendgains an upper bound, because the band has two edges:Far above a word the boundary test always fires and predicts again, leaving the blend paying its extra store against a whole-word fill that dominates either way. Without the bound the nullable 1024 cases came out 13% behind the original kernel, reproducibly across two runs.
New bench target
run_end_decode_bool_ablation, which A/Bs the kernel structures in one process against bench-local mirrors of the previous kernel and of the append-based splat. Both constants can be re-derived from it by pinning a kernel and re-running the grid.Reviewer notes
#[inline(never)], and that is load-bearing. With both loops inlined into one function, the branching loop lost 20% with its own text unchanged, measured against an in-process anchor that did not move. Expressing the choice as a const generic over a single loop did the same. Only keeping the second loop out of line leaves the first alone. Similarly, the spanning path needs#[inline(always)]and not#[inline]— the crate build leaves it out of line where an identical bench-local copy is inlined, which costs most of commit 2's win.finishmust also mask the last partial word or the paint survives past the logical length wheretrue_countwould read it. Handling the last word by a separate slower path instead was measured and does not pay: the only per-run cost it could remove is the store's bounds check, which an unchecked-store experiment puts inside the noise.RUN_END_THRESHOLD), so very short runs are unreachable. Measured at that floor, always blending is free on the non-nullable path but costs 14% on the nullable one, which is reachable through comparisons pushed into a nullable run-end column. The dispatch is free, so it stays.What APIs are changed? Are there any user-facing changes?
runend_decode_typed_boolchanges signature: run ends as&[E]plus anoffset, instead of animpl Iterator<Item = usize>of already-trimmed ends. No callers outside this crate.No behaviour change. Covered by a differential test against a naive element-at-a-time decode across run lengths 1–300, nullable and not, at offsets 0/1/63/64/100; a separate differential test pinning the blend form at eleven run lengths and three lengths chosen to land on and off the word grid; and word-boundary cases at every alignment.
Run position advances by the clamped run length rather than jumping to the run end, so a malformed array cannot run the output position backwards and overrun the sink. Run ends are only checked for sortedness under
debug_assertions, so a release build does see unsorted ends;decode_bools_non_monotonic_endscovers it. All indexing stays bounds-checked — this is a panic-vs-correct-output question, not a soundness one. The onlyunsafeadded is a containedBufferMut::set_lenper output buffer, after the sink has provably initialised every word.Checks run:
cargo test -p vortex-runend(242 pass),cargo test -p vortex-btrblocks -p vortex-layout -p vortex-file,cargo clippy -p vortex-runend --all-targets --all-features,cargo +nightly fmt --all,RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --document-private-items,cargo test --doc,git diff --check, plusobjdumpon the built binaries to confirm the inlining claims above.