Forward detected AI coding-agent id into XCUITest simulator runs#2431
Draft
ctufts wants to merge 1 commit into
Draft
Forward detected AI coding-agent id into XCUITest simulator runs#2431ctufts wants to merge 1 commit into
ctufts wants to merge 1 commit into
Conversation
Adds test-tooling to verify that outbound Mapbox requests made during an XCUITest simulator run carry an `agent/<id>` User-Agent token when the host machine running xcodebuild is driven by a supported AI coding agent (Claude Code, Codex, Cursor, etc.): - scripts/agent-detect.sh detects the agent from the host shell's environment, using the same allowlist/precedence/sanitized-fallback rules as this project's sibling SDK implementations. - scripts/run-uitests-with-agent-tag.sh runs the ExamplesUITests scheme on a simulator, forwarding the detected id to xcodebuild via TEST_RUNNER_MAPBOX_AGENT. - ExamplesUITests.swift and MapLoadingUITest.swift copy MAPBOX_AGENT from the test runner's environment into XCUIApplication's launchEnvironment before launch, since the app under test does not inherit it automatically. None of this affects a normal app run or a UI test run started any other way: nothing is forwarded unless TEST_RUNNER_MAPBOX_AGENT was explicitly set on the xcodebuild invocation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mapbox wants to measure how much SDK traffic across its client SDKs comes from AI coding agents (Claude Code, Codex, Cursor, and similar tools), by appending an
agent/<id>token to the outboundUser-Agentheader whenever an agent can be detected. This repo doesn't build that header itself - that happens in a shared native HTTP layer this SDK depends on - but verifying the tagging end-to-end on iOS means a detected agent id needs a way to reach the running app process during a test run. This PR adds that path, for XCUITest simulator runs only.scripts/agent-detect.shdetects the AI coding agent (if any) driving the host shell about to invokexcodebuild, using the same allowlist, precedence order, and sanitized fallback rules as this project's sibling SDK implementations of the same detection logic (kept in sync across those SDKs).scripts/run-uitests-with-agent-tag.shruns theExamplesUITestsscheme on a simulator, forwarding the detected id toxcodebuild testvia theTEST_RUNNER_MAPBOX_AGENTenvironment variable, which Xcode automatically strips and exposes asMAPBOX_AGENTto the XCUITest runner process.ExamplesUITests.swiftandMapLoadingUITest.swiftreadMAPBOX_AGENTfromProcessInfo.processInfo.environmentand copy it intoXCUIApplication().launchEnvironmentbefore.launch(), since the app under test doesn't automatically inherit the test runner's environment.A normal app run is unaffected, and a UI test run started any other way (without
TEST_RUNNER_MAPBOX_AGENTset) behaves exactly as before - nothing gets added tolaunchEnvironment. No production SDK code is touched. Actually seeing theagent/<id>token show up server-side still depends on a separate, not-yet-merged change to the shared native HTTP layer's User-Agent construction; this PR just makes sure a detected id can reach the app process once that lands.I ran
scripts/agent-detect.shdirectly against a battery of scenarios: each supported agent's indicator var, precedence when multiple are set at once, theVTCODE/TERM_PROGRAMexact-match cases, safe and unsafe fallback-variable handling (including a header-injection-shaped value), and the "nothing detected" case. All produced the expected output, and that same battery runs as a self-check viascripts/agent-detect.sh --self-test. Both scripts passshellcheckandbash -n. I also parsed and typechecked the modified Swift files standalone withswiftcagainst the iOS Simulator SDK; the only errors reported are pre-existing and unrelated to this change (anXCTAssertTrue/XCTAssertEqualmacro-import limitation specific to typechecking a single file outside the full Xcode project build).What I couldn't verify here: a full
xcodebuild testsimulator run ofExamplesUITests, since it needs Carthage/CocoaPods binary dependency downloads gated on a Mapbox secret token and~/.netrc, pluscarthage/podtooling not available in this environment. A reviewer with a normal local dev setup (perDEVELOPING.md) should runscripts/run-uitests-with-agent-tag.shto confirm the app process actually receivesMAPBOX_AGENT, and later, once the native HTTP layer change lands, that outboundUser-Agentheaders carry theagent/<id>token. Real-device and CI runs weren't exercised either.Test plan
scripts/run-uitests-with-agent-tag.shfrom a machine with an AI coding agent's environment indicator set (e.g. inside a Claude Code session) and confirm the launched app's process environment containsMAPBOX_AGENT.TEST_RUNNER_MAPBOX_AGENTargument is passed and the run behaves as before.User-Agentheaders includeagent/<id>.Pull request checklist:
scripts/agent-detect.sh --self-testcovers the detection logic itself; the XCUITest-side plumbing is exercised by the manual Test plan below since it depends on a real simulator run.## mainheading near the top). Not added yet - open to adding one if maintainers want this test-tooling change logged.v10.[version]release branch fix / enhancement, merge it tomainfirst and then port tov10.[version]release branch. N/A, targetingmaindirectly.PRs must be submitted under the terms of our Contributor License Agreement CLA.