Fixes #30459: dedupe Databricks runs by start_time to avoid bulk status 500

[ulixius9]

Describe your changes:

Fixes #30459

I made the Databricks pipeline connector dedupe job runs by start_time before building the bulk PipelineStatus payload, because OpenMetadata stores a single status per timestamp (entityFQNHash, extension, timestamp) and Databricks' paginated run list returns boundary runs more than once (inclusive start_time_to). Those duplicate-timestamp rows made the server-side bulk upsert fail on PostgreSQL with ERROR: ON CONFLICT DO UPDATE command cannot affect row a second time (HTTP 500), dropping run history for the affected pipelines. The fix skips already-seen start_times in yield_pipeline_status, so each bulk request carries at most one status per timestamp.

Type of change:

• Bug fix

High-level design:

N/A — small change.

Tests:

Use cases covered

• Databricks pipeline ingestion where a job's run history contains repeated runs with the same start_time no longer 500s on /status/bulk; only one status per timestamp is sent.

Unit tests

• I added a unit test for the changed logic.
• Files added/updated: ingestion/tests/unit/topology/pipeline/test_databricks_pipeline.py (test_yield_pipeline_status_deduplicates_run_timestamps).
• The test fails before the fix (duplicate-timestamp statuses pass through) and passes after; the existing test_yield_pipeline_status is unchanged. All 6 tests in the file pass.

Backend integration tests

• Not applicable (no backend API changes).

Ingestion integration tests

• Not applicable (covered by the unit test above).

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

• Root cause confirmed from production server logs: the failing /status/bulk requests raised org.postgresql.util.PSQLException: ERROR: ON CONFLICT DO UPDATE command cannot affect row a second time, and every timestamp in a failing batch appeared 2+ times.

UI screen recording / screenshots:

Not applicable.

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #30459 above.
• I have commented on my code, particularly in hard-to-understand areas.
• I have added tests (unit) and listed them above.
• I have added a test that covers the exact scenario we are fixing.

Greptile Summary

Fixes Databricks pipeline-status bulk ingestion failures by:

• Deduplicating job runs by start_time before constructing the bulk status payload.
• Adding a unit test covering repeated run timestamps.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failures remain within the scope of the follow-up review.

Important Files Changed

Filename	Overview
ingestion/src/metadata/ingestion/source/pipeline/databrickspipeline/metadata.py	Deduplicates Databricks runs by status timestamp before producing the bulk pipeline-status record.
ingestion/tests/unit/topology/pipeline/test_databricks_pipeline.py	Adds regression coverage confirming duplicate timestamps collapse into unique pipeline statuses.

_{Reviews (2): Last reviewed commit: "Merge branch 'main' into lusaka" | Re-trigger Greptile}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

[github-actions]

🔴 Playwright Results — workflow failed

Validated commit 737931fbeae82bf95339e95ba3841908d210bb62 in Playwright run 30101783289, attempt 1.

✅ 105 passed · ❌ 2 failed · 🟡 0 flaky · ⏭️ 3 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 51m 23s

⏱️ Max setup 3m 4s · max shard execution 12m 27s · max shard-job elapsed before upload 20m 55s · reporting 4s

🌐 204.13 requests/attempt · 1.74 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

• Browser traffic was 204.13 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 1.74 per UI scenario (210 boots / 121 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	46	0	0	3	0	0
✅ Shard ingestion-01	28	0	0	0	0	0
🔴 Shard ingestion-02	31	2	0	0	0	0

Genuine Failures (failed on all attempts)

❌ Features/TestSuiteMultiPipeline.spec.ts › TestSuite multi pipeline support (shard ingestion-02)

Error: locator.click: Error: strict mode violation: getByText('Data Observability') resolved to 2 elements:     1) <div class="d-flex justify-between gap-1">Data Observability</div> aka getByTestId('profiler').getByText('Data Observability')     2) <span class="tw:truncate tw:text-xs tw:font-medium tw:text-primary">Data Observability</span> aka getByTestId('asset-health-row-dataObservability').getByText('Data Observability')  Call log: �[2m  - waiting for getByText('Data Observability')�[22m 

❌ Features/TestSuiteMultiPipeline.spec.ts › Edit the pipeline's test case (shard ingestion-02)

Error: locator.click: Error: strict mode violation: getByText('Data Observability') resolved to 2 elements:     1) <div class="d-flex justify-between gap-1">Data Observability</div> aka getByTestId('profiler').getByText('Data Observability')     2) <span class="tw:truncate tw:text-xs tw:font-medium tw:text-primary">Data Observability</span> aka getByTestId('asset-health-row-dataObservability').getByText('Data Observability')  Call log: �[2m  - waiting for getByText('Data Observability')�[22m 

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved

Deduplicates Databricks pipeline runs by start_time in yield_pipeline_status to prevent bulk status 500 errors on PostgreSQL. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud