test(workflow-core): extend small-class coverage and add LargeBinary/OnIceberg specs - #7025
Open
aglinxinyuan wants to merge 1 commit into
Open
test(workflow-core): extend small-class coverage and add LargeBinary/OnIceberg specs#7025aglinxinyuan wants to merge 1 commit into
aglinxinyuan wants to merge 1 commit into
Conversation
…OnIceberg specs Add 42 tests across five classes (no source changes; extends are insertions only): - PartitionInfo: hash-attribute content/order sensitivity in satisfies/merge, the RangePartition always-Unknown merge and companion collapse to UnknownPartition, and the polymorphic JSON contract (type-id tagging + base-type round-trip). - JGitVersionControl: initRepo branch extraction, empty-tree and nested-directory node building, symbolic refs, per-commit content history, binary-safe copies, and staged deletions reported as uncommitted. - Tuple: constructor null guards, positional access, inMemSize growth, and the Builder's missing/duplicate/size-mismatch error paths. - LargeBinary (new spec): URI validation with exact messages, bucket/object-key parsing incl. the empty-path branch, value semantics, the thread-local base-URI constructor, and Jackson @JsonValue/@JsonCreator behavior. - OnIceberg (new spec): expireSnapshots collapsing snapshots while retaining live rows, idempotence, the snapshot-less no-op, and the missing-table error — driven against a local Hadoop catalog on a temp warehouse (no REST/S3 needed).
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7025 +/- ##
============================================
+ Coverage 79.20% 79.23% +0.02%
- Complexity 3786 3794 +8
============================================
Files 1160 1160
Lines 46118 46118
Branches 5114 5114
============================================
+ Hits 36529 36542 +13
+ Misses 7969 7960 -9
+ Partials 1620 1616 -4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 390 | 0.238 | 23,670/39,423/39,423 us | 🔴 +14.3% / 🔴 +149.5% |
| 🟢 | bs=100 sw=10 sl=64 | 792 | 0.483 | 124,165/150,169/150,169 us | 🟢 -7.5% / 🔴 +39.9% |
| 🔴 | bs=1000 sw=10 sl=64 | 919 | 0.561 | 1,085,101/1,166,741/1,166,741 us | 🔴 +5.8% / 🔴 +13.5% |
Baseline details
Latest main a61702f from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 390 tuples/sec | 428 tuples/sec | 787.55 tuples/sec | -8.9% | -50.5% |
| bs=10 sw=10 sl=64 | MB/s | 0.238 MB/s | 0.261 MB/s | 0.481 MB/s | -8.8% | -50.5% |
| bs=10 sw=10 sl=64 | p50 | 23,670 us | 22,743 us | 12,255 us | +4.1% | +93.1% |
| bs=10 sw=10 sl=64 | p95 | 39,423 us | 34,497 us | 15,802 us | +14.3% | +149.5% |
| bs=10 sw=10 sl=64 | p99 | 39,423 us | 34,497 us | 19,008 us | +14.3% | +107.4% |
| bs=100 sw=10 sl=64 | throughput | 792 tuples/sec | 811 tuples/sec | 997.81 tuples/sec | -2.3% | -20.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.483 MB/s | 0.495 MB/s | 0.609 MB/s | -2.4% | -20.7% |
| bs=100 sw=10 sl=64 | p50 | 124,165 us | 118,622 us | 100,690 us | +4.7% | +23.3% |
| bs=100 sw=10 sl=64 | p95 | 150,169 us | 162,357 us | 107,316 us | -7.5% | +39.9% |
| bs=100 sw=10 sl=64 | p99 | 150,169 us | 162,357 us | 113,823 us | -7.5% | +31.9% |
| bs=1000 sw=10 sl=64 | throughput | 919 tuples/sec | 938 tuples/sec | 1,030 tuples/sec | -2.0% | -10.8% |
| bs=1000 sw=10 sl=64 | MB/s | 0.561 MB/s | 0.572 MB/s | 0.629 MB/s | -1.9% | -10.8% |
| bs=1000 sw=10 sl=64 | p50 | 1,085,101 us | 1,063,723 us | 981,213 us | +2.0% | +10.6% |
| bs=1000 sw=10 sl=64 | p95 | 1,166,741 us | 1,102,312 us | 1,027,605 us | +5.8% | +13.5% |
| bs=1000 sw=10 sl=64 | p99 | 1,166,741 us | 1,102,312 us | 1,055,466 us | +5.8% | +10.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,512.53,200,128000,390,0.238,23669.63,39423.39,39423.39
1,100,10,64,20,2525.76,2000,1280000,792,0.483,124165.18,150169.07,150169.07
2,1000,10,64,20,21768.11,20000,12800000,919,0.561,1085100.67,1166741.19,1166741.19
Contributor
There was a problem hiding this comment.
Pull request overview
This PR increases unit test coverage in common/workflow-core by extending existing specs and adding new specs for previously under-tested small classes, aligning with issue #7023’s goal of closing coverage gaps without changing production code.
Changes:
- Expanded
PartitionInfoSpec,TupleSpec, andJGitVersionControlSpecwith additional behavioral/edge-case assertions. - Added a new
LargeBinarySpeccovering URI validation, parsing, value semantics, thread-local base-URI behavior, and Jackson (de)serialization. - Added a new
OnIcebergSpecexercisingexpireSnapshotsbehavior against a local Hadoop-backed Iceberg catalog (file-based warehouse).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/workflow/PartitionInfoSpec.scala | Adds tests for Hash/Range semantics and polymorphic JSON type tagging/round-trips. |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/tuple/TupleSpec.scala | Adds tests for constructor null-guards, positional access, sizing, and builder error paths. |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/tuple/LargeBinarySpec.scala | New spec validating LargeBinary URI rules, parsing, thread-local base URI generation, and JSON behavior. |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/util/dataset/JGitVersionControlSpec.scala | Adds tests for init branch name, empty/deep trees, symref hash resolution, binary-safe reads, and staged deletions. |
| common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/result/iceberg/OnIcebergSpec.scala | New spec validating snapshot expiration behavior and missing-table error behavior using a local Hadoop catalog. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // a fresh thread starts with no base URI, so the generating constructor fails fast | ||
| val outcome = onFreshThread(None)(new LargeBinary()) | ||
| assert(outcome.isLeft) | ||
| assert(outcome.left.exists(_.isInstanceOf[IllegalStateException])) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Adds 42 tests across five
common/workflow-coreclasses (49 -> 91 in these suites). No source changes; the three extended specs are pure insertions.PartitionInfo:satisfies/mergesensitivity to hash-attribute contents and order,RangePartition's always-Unknown merge override and the companion's collapse toUnknownPartitionon an empty attribute list, plus the polymorphic-JSON contract (each subtype tagged with its registered type id, base-type round-trip preserving range bounds).JGitVersionControl:initRepo's branch extraction matching what JGit reports,getRootFileNodeOfCommiton an empty tree and on depth-two nested directories, symbolic refs (HEAD) on both read paths, per-commit content after re-staging a modified file, binary-safe copies through both stream APIs, and a staged deletion reported as uncommitted until committed.Tuple: constructor null guards, positionalgetFieldand its out-of-range failure,inMemSizegrowth, and the Builder's missing-attribute / duplicate-add /addSequentiallysize-mismatch paths.LargeBinary(new spec): URI validation with exact messages (including the case-sensitives3://guard), bucket/object-key parsing with the empty-path branch, value semantics, the thread-local base-URI constructor and its unset failure, and Jackson@JsonValue/@JsonCreatorbehavior.OnIceberg(new spec):expireSnapshotscollapsing three snapshots to one while keeping every live row, idempotence, a no-op on a snapshot-less table, and theNoSuchTableExceptionmessage — driven against a local Hadoop catalog on a tempfile:/warehouse, so no REST/S3 backend is needed.A few defensive branches were left uncovered rather than contrived (unreachable
return nullguards ininitRepo, and a parent-node creation branch the tree walk can't reach); they're listed in the review notes.Any related issues, documentation, discussions?
Closes #7023.
How was this PR tested?
sbt -java-home <jbr-17> "WorkflowCore/testOnly *PartitionInfoSpec *JGitVersionControlSpec *TupleSpec *LargeBinarySpec *OnIcebergSpec"-> 91 succeeded, 0 failed.Test/scalafmtCheck+Test/scalafix --checkclean.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8 [1M context])