Skip to content

Fix rex parent_environ case sensitivity on Windows#2098

Open
thc1006 wants to merge 5 commits into
AcademySoftwareFoundation:mainfrom
thc1006:fix/rex-windows-parent-environ-case-2089
Open

Fix rex parent_environ case sensitivity on Windows#2098
thc1006 wants to merge 5 commits into
AcademySoftwareFoundation:mainfrom
thc1006:fix/rex-windows-parent-environ-case-2089

Conversation

@thc1006

@thc1006 thc1006 commented May 7, 2026

Copy link
Copy Markdown

Fixes #2089

On Windows, environment variable keys are case-insensitive natively. os.environ preserves that contract through Python's os._Environ wrapper, but a plain dict copy of it (or any user-built dict) does not. Until now, ActionManager consumed whatever Mapping the caller passed as-is, so a package whose commands() referenced env.SomeVariable against a parent_environ that held the same variable under a different casing would raise:

Error in commands in package '...':
Referenced undefined environment variable: SomeVariable

Issue #2089 has a clean reproducer that follows exactly this shape: dict(os.environ) is passed to ResolvedContext.get_environ, and a package looks up env.SomeVariable while the dict only has SOMEVARIABLE.

The fix wraps a caller-supplied parent_environ in a small read-only, case-normalized snapshot (_CaseInsensitiveEnvironProxy), gated on platform_.name == "windows". os.environ and None are used as-is (they are already case-insensitive on Windows), so their identity and liveness are preserved. On Linux and macOS a caller-supplied mapping retains the existing identity passthrough. Keys are upper-cased once at construction, i.e. a point-in-time snapshot, which matches how parent_environ is fixed for an executor's lifetime; iteration matches what os.environ yields on Windows, and when the input contains keys that differ only by case the last one encountered wins, mirroring how os.environ collapses duplicates. The proxy also exposes .copy() (returning a plain dict with upper-cased keys) for callers that expect a dict-like parent_environ. The not isinstance(...) guard in the gate keeps wrapping idempotent.

Tests:

  • test_case_insensitive_environ_proxy unit-tests the proxy directly and runs on every platform, so the core normalization logic has real CI coverage: get/contains across casings, missing-key KeyError, .copy(), iteration/length, and last-write-wins collisions.
  • test_parent_environ_case_insensitive_on_windows (Windows-only via @unittest.skipIf) drives the real package-commands entrypoint — env.<MixedCase> — through both execute_function and execute_code via the _test helper, alongside direct parent_environ[...] lookups across casings.
  • test_parent_environ_identity_passthrough_on_non_windows (non-Windows) asserts ex.manager.parent_environ is env to guard against accidental wrapping.

Verified locally on Linux: python -m unittest rez.tests.test_rex → 17 passed + 1 Windows-only skip. The full rez-selftest, flake8, and mypy matrix runs in CI.

Scope: this makes lookups against a caller-supplied parent_environ case-insensitive on Windows — the exact #2089 reproducer. Variables set within rex code (ActionManager.environ) are stored verbatim and are not case-folded; that is a separate, pre-existing gap and intentionally out of scope here — tracked in #2164.

Reported by @nrusch. Independent diagnosis previously posted in #2093 (closed because the author could not satisfy the EasyCLA/DCO requirement, not for technical reasons).

Disclosure: Claude Code (Opus 4.8) used for adversarial review, edge-case analysis, test design, and implementation of the review feedback.

@thc1006
thc1006 requested a review from a team as a code owner May 7, 2026 05:12
@linux-foundation-easycla

linux-foundation-easycla Bot commented May 7, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: thc1006 / name: thc1006 (f3db9fb)

@thc1006

thc1006 commented May 7, 2026

Copy link
Copy Markdown
Author

Closing+reopening to retrigger Read the Docs (initial build failed during clone, 2s duration, generic clone error — fork branch is public and accessible).

@thc1006 thc1006 closed this May 7, 2026
@thc1006 thc1006 reopened this May 7, 2026
On Windows env-var keys are case-insensitive natively. os.environ
preserves that contract via os._Environ, but a plain dict copy of it
(or any user-built dict) does not. Until now, ActionManager consumed
whatever Mapping the caller passed as-is, so a package commands()
that referenced env.SomeVariable against a parent_environ holding the
same key under a different case raised RexUndefinedVariableError.

Wrap the caller-supplied parent_environ in a small read-only
case-insensitive proxy on the Windows path only. The proxy is module
private; the wrapping is gated on platform_.name == "windows" and is
idempotent so re-entering the gate is a no-op. Linux and macOS take
the existing identity assignment unchanged.

Also adds a Windows-only regression test that mirrors the issue
reproducer and asserts both direct lookup and end-to-end rex code
paths against a mixed-case parent_environ.

Closes AcademySoftwareFoundation#2089

Reported by @nrusch.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006
thc1006 force-pushed the fix/rex-windows-parent-environ-case-2089 branch from f3db9fb to fa436d7 Compare May 7, 2026 05:21
@thc1006

thc1006 commented May 7, 2026

Copy link
Copy Markdown
Author

Status note for reviewers:

  • The six core GitHub Actions workflows (tests, flake8, mypy, copyright, installation, benchmark) are queued in action_required state, awaiting maintainer approval per the first-time-contributor policy. Nothing to do on my side.
  • The Read the Docs failure is git fetch origin pull/2098/head returning fatal: couldn't find remote ref pull/2098/head. The GitHub git API for this PR currently exposes only refs/pull/2098/merge; refs/pull/2098/head is missing from matching-refs/pull/2098, while other open PRs (e.g. Allow building with cmake on Windows #2090, Fix shell auto-completion #2091) have both refs as expected. An amend force-push was attempted to nudge GitHub into recreating the head ref but it has not reappeared. This appears to be a GitHub-side propagation glitch independent of the diff. Approving the workflows may incidentally fix it; if not, an RTD admin rebuild should clear it.
  • DCO and EasyCLA are both green.

Local pre-push validation: pytest src/rez/tests/test_rex.py is green (15 pass + 1 Windows-only skip); full rez-selftest on a hermetic Python 3.11.14 container reports Ran 259 tests in 24.997s, OK (skipped=10); flake8 src/rez/rex.py src/rez/tests/test_rex.py is clean; mypy | mypy-baseline filter --ignore-categories=note --allow-unsynced reports no new violations above the pinned baseline.

Copilot AI review requested due to automatic review settings May 7, 2026 06:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@maxnbk maxnbk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for taking a long time to get to this review.

I've commented on a few specific items.

More generally, _CaseInsensitiveEnvironProxy could have a .copy() method added, which would be defensive against future code calling self.parent_environ.copy() to return a plain dict with uppercased keys.

ActionManager.__init__ docstring could document the Windows wrapping behavior.

And last, I feel it would be useful to consider a non-Windows identity passthrough test. i.e. assert ex.manager.parent_environ is env on non-Windows to guard against accidental wrapping. With that in mind, the test skip could be avoided and just assert differently for different host types, or be a different test skipping windows hosts, either way.

If you are using an LLM to produce this PR, please add a line to the PR description disclosing the usage, i.e. "Disclosure: used for (whichever things it was used for.)

Comment thread src/rez/tests/test_rex.py Outdated
Comment thread src/rez/rex.py Outdated
thc1006 added 2 commits July 21, 2026 19:17
- Add _CaseInsensitiveEnvironProxy.copy() returning a plain dict with
  upper-cased keys, for callers that expect a dict-like parent_environ.
- Pass os.environ through unwrapped (it is already case-insensitive on
  Windows), preserving its identity and liveness.
- Document the Windows wrapping behaviour in ActionManager.__init__ and
  clarify that the proxy is a construction-time snapshot, not a live view.
- Drop the isinstance(key, str) guard in __contains__ so the proxy
  behaves like a normal mapping.
- Tests: add a platform-independent unit test for the proxy, a
  non-Windows identity passthrough test, and switch the Windows
  regression test to @unittest.skipIf and the real env.<MixedCase>
  entrypoint via _test (execute_function + execute_code).

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The Windows fix normalizes lookups against a caller-supplied
parent_environ only. ActionManager.environ (variables set within rex
code) is stored verbatim and is not case-folded, which is a separate,
pre-existing concern now tracked in AcademySoftwareFoundation#2164. Tighten the __init__ and
proxy docstrings to say "lookups against parent_environ are
case-insensitive" instead of the broader "match native Windows
semantics", and note the boundary next to self.environ.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006
thc1006 force-pushed the fix/rex-windows-parent-environ-case-2089 branch from 1b6f0ce to c0b42f8 Compare July 21, 2026 11:17

@maxnbk maxnbk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests failed, I believe it is because rex env is shadowing your variables. I am guessing something like this might fix it. GitHub UI is misbehaving so I wasn't able to apply it as a suggestion to click apply for:

         """Regression for #2089.
         ...
         """
-        env = {"SomeVariable": "covfefe"}
-        ex = self._create_executor(env)
+        parent_env = {"SomeVariable": "covfefe"}
+        ex = self._create_executor(parent_env)

         # any casing of the key resolves
         self.assertEqual(ex.manager.parent_environ["SomeVariable"], "covfefe")
         ...

         # end-to-end via the real package-commands entrypoint
         def _rex() -> None:
             env.RESULT = env.SOMEVARIABLE  # noqa: F821 - rex builtin

         self._test(
             func=_rex,
-            env={"SomeVariable": "covfefe"},
+            env=parent_env,
             expected_actions=[Setenv("RESULT", "covfefe")],
             expected_output={"RESULT": "covfefe"},
         )

(It may be wise to check the tests for any similar pattern and update those just on the off-chance someone introduces rex in any of those tests in the future and the same gotcha resurfaces too)

@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.33%. Comparing base (dc6b3a1) to head (c84ef86).
⚠️ Report is 52 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2098      +/-   ##
==========================================
+ Coverage   60.03%   61.33%   +1.29%     
==========================================
  Files         164      164              
  Lines       20557    20586      +29     
  Branches     3571     3578       +7     
==========================================
+ Hits        12342    12627     +285     
+ Misses       7347     7088     -259     
- Partials      868      871       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The Windows regression test declared a local `env` dict, so the nested
rex function closed over it (LOAD_DEREF) and shadowed the rex `env`
builtin that execute_function injects into globals -- `env.SOMEVARIABLE`
then hit a plain dict and failed on Windows CI. Rename the caller dict to
`parent_env` in both new tests so the closure no longer shadows `env`
(thanks @maxnbk for the diagnosis).

Also address the Windows-only coverage gap codecov flagged:
- test_parent_environ_os_environ_passthrough: covers the
  `parent_environ is os.environ` identity branch.
- test_parent_environ_case_insensitive_windows_mocked: mocks
  platform_.name so the case-insensitive wrapping branch (and the full
  env.<MixedCase> end-to-end) is exercised on non-Windows CI too, giving
  the Windows path real coverage and off-Windows validation.

Audited the rest of test_rex.py: no other test declares a local `env`;
they all pass `env=` as a kwarg to _test, so none shadow the builtin.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006

thc1006 commented Jul 23, 2026

Copy link
Copy Markdown
Author

Thanks for the diagnosis — that was exactly it. execute_function rebuilds the function with closure=func.__closure__, so once the test declared a local env dict the nested _rex captured it as a closure cell (LOAD_DEREF) and shadowed the rex env builtin injected into globals — env.SOMEVARIABLE then hit a plain dict. execute_code was fine (fresh compile from source), which is why only the function path failed.

Fixed in 1c3299e: renamed the caller dict to parent_env in both new tests, so _rex's env is a plain global again and resolves to the rex builtin. Added a short comment on the test flagging the gotcha.

Per your future-proofing note, I audited the rest of test_rex.py: no other test declares a local env — they all pass env= as a kwarg to _test, so none can shadow the builtin.

I also closed the codecov gap on the Windows-only branch (it's skipped on the Linux/macOS jobs):

  • test_parent_environ_os_environ_passthrough — covers the parent_environ is os.environ identity branch.
  • test_parent_environ_case_insensitive_windows_mocked — mocks platform_.name so the case-insensitive wrapping branch and the full env.<MixedCase> end-to-end run on every host. As a bonus that validates the Windows logic off-Windows, which is what would have caught this shadowing bug locally.

@maxnbk maxnbk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It LGTM now. Thank you for making the requested changes @thc1006 . I marked Approve (from me) and put this into "Next" milestone for merge in the next release. No further action needed unless some other reviewer requests changes.

@maxnbk maxnbk added this to the Next milestone Jul 23, 2026
Comment thread src/rez/rex.py Outdated
Comment thread src/rez/rex.py Outdated
Comment thread src/rez/rex.py Outdated
Comment thread src/rez/rex.py Outdated
Comment thread src/rez/rex.py
return self.interpreter.get_key_token(key)


class _CaseInsensitiveEnvironProxy(Mapping):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class _CaseInsensitiveEnvironProxy(Mapping):
class _CaseInsensitiveEnvironProxy(Mapping[str, str]):

?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tl;dr: happy to take the Mapping[str, str] change, but it should use typing.Mapping to stay 3.8-safe. Separately, the failing Ruff check looks like a pre-existing issue on main, not something in this PR.

On the base class: Mapping here is imported from collections.abc, and using one of those as a subscripted base (class X(Mapping[str, str])) only works on Python 3.9+. On 3.8 it raises TypeError at import, and we still support 3.8 (python_requires >= 3.8). The other parametrized bases in the repo, like AttrDictWrapper(MutableMapping[str, Any]) and PackageOrderList(List[...]), go through typing for that reason. I can switch this to from typing import Mapping and apply Mapping[str, str], or leave it bare, whichever you prefer.

On the Ruff failure: it's ISC004 at src/rez/utils/platform_.py:524 (the PowerShell command list), which this branch doesn't touch. It reproduces on a clean main (4946f9de) with ruff 0.16.0, the newest release matching >= 0.15.0 that ruff-action pulls in. So it reads as the new ISC004 rule catching an existing line rather than anything from here. I'm happy to fold the one-line fix into this PR if that unblocks things, otherwise it may fit better as its own change on main.

Thanks for the sphinx pass too, the :data:/:class: refs read a lot cleaner.

Co-authored-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>
Signed-off-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rex environment lookups do not respect platform case-sensitivity rules

4 participants