refactor(pyamber): replace PyFilesystem2 with tempfile and unpin setuptools

[aicam]

What changes were proposed in this PR?

Implements the migration planned in #6917: ExecutorManager drops fs (PyFilesystem2) for tempfile + shutil, and setuptools is unpinned. The issue covers the rationale, the before/after shape, the required footprint, and the alternatives that were ruled out — this description only covers what it doesn't.

An extra fix in the same code path. load_executor_definition now calls importlib.invalidate_caches() before importing the file it just wrote. The import system caches a directory listing per sys.path entry and refreshes it only when the directory mtime changes, so a second UDF written into the same tmp directory within one mtime tick can be invisible to the finder and raise ModuleNotFoundError: No module named 'udf-vN'.

This is not introduced here — checking out master's executor_manager.py and test_executor_manager.py with fs==2.4.16 installed reproduces it locally (4 TestUpdateExecutor failures in the same environment). It is fixed here because the write-then-import path is exactly what's being rewritten, and a reviewer running the suite locally would otherwise attribute the flake to this change.

One planned item turned out to be a no-op. The issue asks the migration PR to also revert the two warning-filter hunks from #6880. #6880 was closed unmerged, so there is nothing on master to revert — amber/pyproject.toml and texera_run_python_worker.py carry no pkg_resources filter and are untouched here. No other deviation from the plan in #6917.

Any related issues, documentation, discussions?

Closes #6917. Supersedes #6911 (the setuptools v83 bump — closed, it could not merge while fs was present) and #6412 (setuptools<82, the interim bound). Original pin: #4199, re-affirmed by the #6110 pin audit.

How was this PR tested?

CI is green, including pyamber on 3.11 / 3.12 / 3.13 and amber-integration on both ubuntu and macos.

Locally, on Python 3.12 in a clean venv built from amber/requirements.txt + amber/dev-requirements.txt with proto bindings generated via bin/python-proto-gen.sh — import fs in that venv raises ModuleNotFoundError, confirming nothing reaches it:

• pytest -m "not integration" → 862 passed, 1 deselected, 1 xfailed. This exercises ExecutorManager directly and end to end through UDF loading (pytexera/udf/, test_initialize_executor_handler.py, test_update_executor_handler.py, runnables/test_main_loop.py).
• test_executor_manager.py run 5× consecutively → 23 passed each time, confirming the invalidate_caches fix is stable (the same file fails 3–4 cases per run without it, on master as well).
• ruff check and ruff format --check over src/main/python src/test/python → clean.
• License drift, the same check CI runs: pip-licenses over the runtime closure + ./bin/licensing/check_binary_deps.py --ignore-transitive-version python → OK: 109 Python packages match LICENSE-binary. Running that same package list against master's LICENSE-binary-python reports exactly STALE: appdirs==1.4.4, fs==2.4.16 — i.e. the removals here are precisely what's required, and nothing more.
• Dependency-closure diff (pip resolution, py3.12, requirements.txt + operator-requirements.txt) between master and this branch: exactly fs==2.4.16 and appdirs==1.4.4 removed, nothing added, six retained, setuptools 80.10.2 → 81.0.0 via torch.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[aglinxinyuan]

Can you check if it's same as #6928?

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @Ma77Ball
  You can notify them by mentioning @Ma77Ball in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.96%. Comparing base (49f9e2c) to head (309a28b).

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #6946   +/-   ##
=========================================
  Coverage     78.96%   78.96%           
  Complexity     3798     3798           
=========================================
  Files          1161     1161           
  Lines         46084    46085    +1     
  Branches       5110     5110           
=========================================
+ Hits          36388    36389    +1     
  Misses         8076     8076           
  Partials       1620     1620           

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.00% <ø> (ø)		Carriedforward from 49f9e2c
agent-service	76.76% <ø> (ø)		Carriedforward from 49f9e2c
amber	72.23% <ø> (ø)		Carriedforward from 49f9e2c
computing-unit-managing-service	20.49% <ø> (ø)		Carriedforward from 49f9e2c
config-service	66.66% <ø> (ø)		Carriedforward from 49f9e2c
file-service	67.21% <ø> (ø)		Carriedforward from 49f9e2c
frontend	82.91% <ø> (ø)		Carriedforward from 49f9e2c
notebook-migration-service	78.94% <ø> (ø)		Carriedforward from 49f9e2c
pyamber	95.38% <100.00%> (+<0.01%)	⬆️
workflow-compiling-service	55.14% <ø> (ø)		Carriedforward from 49f9e2c

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

✅ No material benchmark regressions detected

🟢 2 better · 🔴 0 worse · ⚪ 13 noise (<±5%) · 0 without baseline

Compared against main 49f9e2c benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
⚪	bs=10 sw=10 sl=64	543	0.331	17,682/25,760/25,760 us	⚪ within ±5% / 🔴 +58.7%
🟢	bs=100 sw=10 sl=64	1,197	0.731	84,144/94,151/94,151 us	🟢 -8.5% / 🟢 +22.9%
⚪	bs=1000 sw=10 sl=64	1,419	0.866	708,614/754,598/754,598 us	⚪ within ±5% / 🟢 +41.3%

Baseline details

Latest main 49f9e2c from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	543 tuples/sec	552 tuples/sec	767.9 tuples/sec	-1.6%	-29.3%
bs=10 sw=10 sl=64	MB/s	0.331 MB/s	0.337 MB/s	0.469 MB/s	-1.8%	-29.4%
bs=10 sw=10 sl=64	p50	17,682 us	18,348 us	12,502 us	-3.6%	+41.4%
bs=10 sw=10 sl=64	p95	25,760 us	24,724 us	16,234 us	+4.2%	+58.7%
bs=10 sw=10 sl=64	p99	25,760 us	24,724 us	18,919 us	+4.2%	+36.2%
bs=100 sw=10 sl=64	throughput	1,197 tuples/sec	1,217 tuples/sec	974.8 tuples/sec	-1.6%	+22.8%
bs=100 sw=10 sl=64	MB/s	0.731 MB/s	0.743 MB/s	0.595 MB/s	-1.6%	+22.9%
bs=100 sw=10 sl=64	p50	84,144 us	82,744 us	102,449 us	+1.7%	-17.9%
bs=100 sw=10 sl=64	p95	94,151 us	102,894 us	108,652 us	-8.5%	-13.3%
bs=100 sw=10 sl=64	p99	94,151 us	102,894 us	116,310 us	-8.5%	-19.1%
bs=1000 sw=10 sl=64	throughput	1,419 tuples/sec	1,410 tuples/sec	1,004 tuples/sec	+0.6%	+41.3%
bs=1000 sw=10 sl=64	MB/s	0.866 MB/s	0.86 MB/s	0.613 MB/s	+0.7%	+41.3%
bs=1000 sw=10 sl=64	p50	708,614 us	708,011 us	999,606 us	+0.1%	-29.1%
bs=1000 sw=10 sl=64	p95	754,598 us	752,207 us	1,046,770 us	+0.3%	-27.9%
bs=1000 sw=10 sl=64	p99	754,598 us	752,207 us	1,076,937 us	+0.3%	-29.9%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,368.59,200,128000,543,0.331,17682.42,25759.69,25759.69
1,100,10,64,20,1670.49,2000,1280000,1197,0.731,84144.16,94150.73,94150.73
2,1000,10,64,20,14090.30,20000,12800000,1419,0.866,708614.04,754597.96,754597.96

[Yicong-Huang]

@aicam please mark this as a draft as we are discussing on the solution. Thanks

[chenlica]

@aicam Your PR description is presumably generated by AI, and it's verbose and hard to read. Please revise it to make it more human readable.

[aicam]

@aicam Your PR description is presumably generated by AI, and it's verbose and hard to read. Please revise it to make it more human readable.

Done, I removed repeated information from issue and file changes.