.NET: Resilient long-running (crash-recoverable) Foundry hosted agents - #7370
Draft
rogerbarreto wants to merge 18 commits into
Draft
.NET: Resilient long-running (crash-recoverable) Foundry hosted agents#7370rogerbarreto wants to merge 18 commits into
rogerbarreto wants to merge 18 commits into
Conversation
Point the AgentServer package versions at the drop that carries the resilient / checkpointing / steering surface (Core beta.27, Invocations beta.6, Responses beta.7) and add the agentserver-preview-local source mapped to Azure.AI.AgentServer.*.
Document, in plain language, when durable long-running hosting applies (background requests only; workflow or safe-to-redo single agent), the recovery contract, the resilience gate, host-wide single registration, the agent-registration readiness health check, session-store durability, and the two focused samples.
Lean opt-in wiring and the crash-recovery save cadence, all inert unless resilience is turned on: - AddFoundryResponses flows Action<ResponsesServerOptions> straight to the SDK (no wrapper options type). - Single-registration guard: the Responses server is host-wide (one per process). A prior registration is detected via the ResponseHandler it always registers; the single-agent AddFoundryResponses(agent) extension throws an actionable message if combined or repeated, and the parameterless overload is a no-op on repeat. - Agent-registration readiness health check: unhealthy until at least one AIAgent is registered (keyed or default), scanning service descriptors so it never constructs an agent. - Resilience gate: the handler reads IOptions<ResponsesServerOptions> for ResilientBackground; ShouldPersistForResilience requires background and store. - Save cadence: on a resilient turn (or IsRecovery), persist the session at each completed output item so a mid-turn crash leaves the last workflow step on disk; on graceful shutdown, ExitForRecoveryAsync instead of emitting incomplete. - Tests for the guard and the health check.
- Hosted-Workflow-Resilient: the translation-chain workflow hosted with ResilientBackground enabled, so a background response survives a container crash and resumes from the workflow's last completed step. README has a local crash-and-recover walkthrough. - Hosted-Steering: a chat agent hosted with SteerableConversations enabled, so a follow-up input for an in-progress conversation is queued instead of rejected. Both opt in with a single option (o => o.ResilientBackground/SteerableConversations = true), are off by default, registered in agent-framework-dotnet.slnx, build clean, and answer GET /readiness with 200 Healthy locally.
On a crash-recovery re-invocation (context.IsRecovery), align the handler with the SDK's resilient pattern so a resumed background response reaches a terminal state instead of stalling: - Seed the event stream from the last durable snapshot (context.PersistedResponse) instead of the request, so the resumed stream continues from the output items already emitted before the crash. The output-index allocator advances to PersistedResponse.Output.Count and re-emitted items do not collide with the already-durable slots. - Skip re-injecting the original input on recovery. The persisted session already carries the progress (a workflow hosted as an agent resumes from its last checkpoint, restored by GetSessionAsync), so re-injecting would enqueue a duplicate turn on top of the resumed state and the run would never terminate. Both changes are gated on IsRecovery, so a normal (non-recovery) turn is unchanged.
A workflow checkpoint records each step by its executor id, and an agent-backed step derives that id from the agent's Id (and Name). By default an agent gets a fresh random Id per process, so after a crash the restarted process would rebuild the workflow with different ids and the saved checkpoint would no longer match, failing the resume with "checkpoint is not compatible with the workflow". Give each translation agent a fixed Id and Name so the rebuilt workflow is identical across restarts and the checkpoint resumes cleanly. A code comment spells out why this matters for resilient workflows.
Add the minimal, non-public hooks the Foundry resilient-hosting readiness check needs to detect, ahead of time, a workflow whose agent-backed steps would not survive a crash-recovery resume: - AIAgent.HasAutoGeneratedId (internal): true when no explicit id was supplied (IdCore is null), so Id falls back to a per-process random value. This is the authoritative signal: a caller-supplied id that happens to be a GUID is correctly reported as stable. Exposed to the hosting package via a new InternalsVisibleTo entry on the abstractions assembly. - WorkflowHostAgent.GetService now returns the hosted Workflow for an unkeyed Workflow service request, so a host can inspect the workflow's executor ids (and their backing agents) without constructing a session. Additive, with no change to existing behaviour. WorkflowHostAgentTests covers the GetService enabler.
A workflow hosted as an agent resumes after a crash by matching its checkpoint to the rebuilt workflow by executor id. An agent-backed step derives that id from the agent's id, so an agent with an auto-generated id produces a different id on every start and the resume fails deep in the engine with "the specified checkpoint is not compatible with the workflow". Add a /readiness health check that, only while ResilientBackground is on, inspects every registered workflow-host agent (via GetService<Workflow>) and reports Unhealthy with an actionable message when any agent-backed step has an auto-generated id (AIAgent.HasAutoGeneratedId). This turns a late, cryptic resume failure into an early not-ready signal. It reads already-constructed agent instances from the binding raw value, so a caller-supplied GUID id is correctly treated as stable, and it never creates a session. On a non-resilient host it is a no-op. Because the check reads abstractions internals via InternalsVisibleTo, the hosting assembly now consumes the shared Throw and DiagnosticIds sources from the abstractions assembly instead of injecting its own copies, matching how other projects consume a referenced assembly's shared sources.
Use options: new() to set each agent's stable Id and Name (the named argument picks the options overload so the target-typed new() is unambiguous), and add a README note explaining why resilient workflows need stable executor ids.
Record in the .NET AGENTS.md that a test file is named after its system under test (the type tested), and that a single file should not mix multiple systems under test.
The non-recovery history-replay guard treated a session as a resume when its state bag held any entry. A hosted session is stamped once, on its first turn, with a write-once HostedSessionContext, which makes the state bag non-empty from the first turn, so counting entries could make a brand new conversation look like a resume and skip loading the history of a conversation the platform resumed by id. Decide instead from a specific marker captured at the right time: whether the session already carried a HostedSessionContext before this turn stamped one. A session the store loaded from a prior turn has it; a freshly created session does not. This reflects whether a prior turn established the session without relying on how many state-bag entries happen to exist. In local (non-hosted) use no context is stamped, so a fresh turn loads history as before.
State the requirement directly: for resilient workflows every workflow step agent must have an explicit id for crash-recovery resume, and list the step agents to fix.
Final pass so the ADR reflects what shipped: the recovery bridge seeds from PersistedResponse and skips input re-injection; progress is persisted per output item through the existing FileSystemAgentSessionStore rather than a separate resilient store type; and resilient workflows require stable executor ids, backed by the internal AIAgent.HasAutoGeneratedId signal, the WorkflowHostAgent Workflow service, and a second readiness health check. Implementation-plan items done in this work are marked done and the Python-to-.NET mapping is corrected.
A workflow hosted as an agent advances on its own thread, so serializing the session at each completed output item (the resilient checkpoint cadence) can race the running workflow's in-memory checkpoint writes and throw "Collection was modified". Because that save sat in the streaming loop's finally-only try, the exception escaped the handler, the orchestrator logged "Handler failed", and the response was left stuck in_progress forever. On a crash-recovery re-invocation the resumed workflow drains its buffered supersteps in a burst, so the race was effectively guaranteed and recovery never completed. Wrap the incremental save in a try/catch that logs and continues. It is a pure durability optimization: the serialize throws before any file is written, so a failed attempt leaves the previously persisted session intact, and the authoritative save in the finally block (after the stream drains and the workflow is quiescent) persists the final consistent state. A transient snapshot race must never abort the turn. Verified with a local crash-and-recover cycle: a run crashed mid-progress now resumes and reaches status=completed with no "Handler failed" in the restart, where before it stayed in_progress.
Drive the handler with a fake agent that records the messages it receives and a fake session store, so the crash-recovery semantics can be asserted without a real model, a real process crash, or timing: - On a recovery re-invocation the handler injects no input (the restored session drives the resume), while a fresh turn feeds the request input to the agent. - A mid-stream session save that throws (the serialize race) is swallowed and the turn still reaches a completed terminal event instead of escaping as a handler failure.
rogerbarreto
temporarily deployed
to
github-app-auth
July 28, 2026 18:44 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
rogerbarreto
temporarily deployed
to
github-app-auth
July 28, 2026 18:44 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
rogerbarreto
temporarily deployed
to
github-app-auth
July 28, 2026 18:44 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
agent-framework-automation
Bot
added
documentation
rogerbarreto
had a problem deploying
to
integration
July 28, 2026 18:45 — with
GitHub Actions
Failure
GitHub Actions
Failure
Contributor
There was a problem hiding this comment.
Pull request overview
Adds opt-in support in the .NET Foundry hosting layer for resilient long-running background Responses (crash recovery) and steerable conversations, aligning behavior with the existing Python reference while keeping the default behavior unchanged unless explicitly enabled.
Changes:
- Plumbs
ResponsesServerOptionsconfiguration throughAddFoundryResponses(...)and adds single-registration guarding + readiness health checks (agent registration + stable workflow executor IDs when resilience is enabled). - Updates
AgentFrameworkResponseHandlerto support recovery semantics (IsRecovery+PersistedResponse) and incremental best-effort session persistence at output-item boundaries. - Adds ADR 0032 plus focused samples and unit tests covering the new resilience/readiness behaviors.
Reviewed changes
Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/WorkflowHostAgentTests.cs | Verifies WorkflowHostAgent.GetService<Workflow>() exposes the hosted workflow and returns null for plain agents. |
| dotnet/tests/Microsoft.Agents.AI.Foundry.Hosting.UnitTests/ServiceCollectionExtensionsTests.cs | Adds tests for server single-registration behavior and readiness health check registration. |
| dotnet/tests/Microsoft.Agents.AI.Foundry.Hosting.UnitTests/ResilientWorkflowExecutorIdHealthCheckTests.cs | Covers readiness failure/success cases for resilient workflow executor ID stability detection. |
| dotnet/tests/Microsoft.Agents.AI.Foundry.Hosting.UnitTests/AgentRegistrationHealthCheckTests.cs | Validates readiness behavior when no agent is registered and descriptor-based detection for keyed/default agents. |
| dotnet/tests/Microsoft.Agents.AI.Foundry.Hosting.UnitTests/AgentFrameworkResponseHandlerResilienceTests.cs | Deterministic tests for recovery input handling, persisted-response seeding, and mid-stream save failure tolerance. |
| dotnet/src/Microsoft.Agents.AI.Workflows/WorkflowHostAgent.cs | Extends GetService to return hosted Workflow for inspection without constructing a session. |
| dotnet/src/Microsoft.Agents.AI.Foundry.Hosting/ServiceCollectionExtensions.cs | Adds ResponsesServerOptions configuration callback, enforces single server registration, and registers readiness health checks. |
| dotnet/src/Microsoft.Agents.AI.Foundry.Hosting/ResilientWorkflowExecutorIdHealthCheck.cs | Implements readiness health check validating stable executor IDs for resilient workflow hosting. |
| dotnet/src/Microsoft.Agents.AI.Foundry.Hosting/Microsoft.Agents.AI.Foundry.Hosting.csproj | Adjusts packaging/build properties for the hosting project. |
| dotnet/src/Microsoft.Agents.AI.Foundry.Hosting/AgentRegistrationHealthCheck.cs | Adds readiness health check to fail fast when no agent is registered. |
| dotnet/src/Microsoft.Agents.AI.Foundry.Hosting/AgentFrameworkResponseHandler.cs | Implements recovery-aware stream seeding, recovery input handling changes, and incremental checkpoint persistence for resilience. |
| dotnet/src/Microsoft.Agents.AI.Abstractions/Microsoft.Agents.AI.Abstractions.csproj | Exposes internals to Foundry.Hosting to enable readiness checks without expanding public API. |
| dotnet/src/Microsoft.Agents.AI.Abstractions/AIAgent.cs | Adds internal HasAutoGeneratedId signal to detect unstable agent IDs. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Workflow-Resilient/README.md | Documents resilient background responses and crash-recovery workflow behavior with local walkthrough. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Workflow-Resilient/Program.cs | Sample showing a resilient hosted workflow with explicit stable agent IDs. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Workflow-Resilient/HostedWorkflowResilient.csproj | Sample project wiring for resilient workflow hosting. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Workflow-Resilient/Dockerfile.contributor | Contributor container build path for ProjectReference-based sample. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Workflow-Resilient/Dockerfile | End-user Dockerfile for the resilient workflow sample. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Workflow-Resilient/agent.yaml | Sample hosted agent definition (container protocol). |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Workflow-Resilient/agent.manifest.yaml | Sample manifest for initializing/deploying the resilient workflow template. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Workflow-Resilient/.env.example | Sample environment configuration for local runs. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Steering/README.md | Documents steerable conversations behavior and usage with Responses protocol. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Steering/Program.cs | Sample showing steerable conversations enabled via ResponsesServerOptions. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Steering/HostedSteering.csproj | Sample project wiring for steerable hosting scenario. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Steering/Dockerfile.contributor | Contributor container build path for ProjectReference-based sample. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Steering/Dockerfile | End-user Dockerfile for the steering sample. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Steering/agent.yaml | Sample hosted agent definition (container protocol). |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Steering/agent.manifest.yaml | Sample manifest for initializing/deploying the steering template. |
| dotnet/samples/04-hosting/FoundryHostedAgents/responses/Hosted-Steering/.env.example | Sample environment configuration for local runs. |
| dotnet/nuget.config | Adds an additional package source and source mapping configuration. |
| dotnet/Directory.Packages.props | Updates Azure.AI.AgentServer.* package versions to betas carrying the resilience surface. |
| dotnet/AGENTS.md | Codifies test-file naming conventions (test file named after SUT; keep tests for a type together). |
| dotnet/agent-framework-dotnet.slnx | Adds the new samples to the solution. |
| docs/decisions/0032-foundry-hosting-resilient-long-running-agents.md | ADR documenting the resilience/steering design, recovery contract, and readiness safeguards. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated Code Review
Reviewers: 5 | Confidence: 89%
✓ Correctness
The PR adds opt-in resilient long-running agent support. The logic is sound and well-tested. The primary issue is a development artifact in nuget.config: a Windows-specific local package source path (
C:\local_packages) that will cause warnings or failures in CI and on non-Windows machines. A secondary concern is passing a potentially-cancelled cancellation token toExitForRecoveryAsyncduring shutdown.
✓ Security Reliability
The most critical issue is a local developer package source (
C:\local_packages) committed to nuget.config, which is a supply-chain risk on shared machines and will cause restore warnings/failures on CI (Linux). The resilience implementation itself is well-structured with appropriate gating and error handling. The mid-stream save's catch-all for non-cancellation exceptions is reasonable given the documented serialize-race scenario. TheExitForRecoveryAsynccall during shutdown uses the original cancellation token which could already be in a cancelling state, though this is a lower-confidence concern.
✓ Test Coverage
The PR adds good focused tests for resilience health checks, recovery semantics, and registration guard logic. However, there are notable test coverage gaps: the core
ExitForRecoveryAsyncshutdown-deferral path is untested, thefoundry-resilient-workflow-idshealth check registration is not verified end-to-end (unlike thefoundry-agent-registrationcheck which has a test), and there is no test verifying that non-resilient turns (Background=false or Store=false) correctly skip incremental persistence. Additionally, the nuget.config commits a hardcoded Windows-local package source that will break CI/restore on non-Windows environments.
✓ Failure Modes
The PR introduces well-tested crash-recovery semantics with properly gated, best-effort mid-stream saves and a clear recovery contract. The primary operational concern is a committed development artifact in nuget.config that adds a Windows-specific local package source (
C:\local_packages) which will not exist in CI or for other contributors, potentially blocking builds if the referenced Azure.AI.AgentServer beta versions are not yet on nuget.org.
✓ Design Approach
I found two design gaps in the new resilience registration path. The resilient-workflow readiness check only inspects prebuilt agent instances, so it silently misses workflow agents registered through keyed DI factories even though that registration style is already used elsewhere in this repo; that defeats the PR's stated goal of surfacing unstable executor IDs before traffic is routed. Separately, repeated parameterless AddFoundryResponses calls silently discard any later ResponsesServerOptions configuration, which can leave resilience or steering disabled even when the caller explicitly opts in on the second call.
Suggestions
- Add a test verifying that when shutdown is detected on a resilient turn,
ExitForRecoveryAsyncis called and no terminal event is emitted (the core crash-recovery deferral path at AgentFrameworkResponseHandler.cs:490-494 is currently untested).
Automated review by rogerbarreto's agents
…authoritative resume detection GetSessionAsync was a hidden get-or-create that always returned a usable session, so the handler could not tell a resumed conversation from a fresh one and inferred resume from HostedSessionContext, which is only stamped on hosted turns. That regressed local (non-hosted) resumes, replaying history over restored state. GetSessionAsync now returns null when nothing is stored (a plain lookup, never creating), which is the authoritative resume signal in both local and hosted runs and for workflows and chat agents. A new virtual GetOrCreateSessionAsync keeps the convenience path. The handler now derives isResume from whether the store actually loaded a session.
…covers factory registrations The readiness health check read AIAgent registrations straight from the service descriptors, which only exposes agents registered as a concrete instance. Agents registered via a factory (including keyed factory registrations, the documented way to host multiple workflow agents) were never inspected, so their executor ids went unchecked. Resolving from the service provider instead covers both instance and factory registrations. Keys are discovered from the descriptors because a service provider cannot enumerate keyed registrations on its own; agents are singletons so each is constructed at most once and cached.
rogerbarreto
temporarily deployed
to
github-app-auth
July 29, 2026 14:50 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
rogerbarreto
had a problem deploying
to
integration
July 29, 2026 14:50 — with
GitHub Actions
Failure
GitHub Actions
Failure
…ns beta.7, Responses beta.8 Refreshed local preview drop from the long-running-agents-private-preview main branch. Foundry.Hosting builds clean and the full unit suite stays green (345/345) on the new surface.
rogerbarreto
temporarily deployed
to
github-app-auth
July 29, 2026 15:03 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
rogerbarreto
had a problem deploying
to
integration
July 29, 2026 15:03 — with
GitHub Actions
Failure
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation & Context
The Foundry Hosted Agents platform can keep a background response running with no client
connected and re-invoke the handler after a container crash or graceful shutdown. This adds
opt-in support for that in the .NET
Microsoft.Agents.AI.Foundry.Hostingpackage, mirroring thePython reference, so a workflow hosted as an
AIAgentcan run as a durable, crash-recoverableFoundry Hosted Agent. Everything is off by default: a host that does not opt in behaves exactly as
before.
Description & Review Guide
What are the major changes?
AddFoundryResponses(o => o.ResilientBackground = true)flows to theAzure SDK options; single-registration guard; agent-registration readiness check.
AgentFrameworkResponseHandler: onIsRecovery, seed thestream from
PersistedResponseand skip re-injecting input; persist the session at eachcompleted output item so a restart resumes from the last step. The mid-stream save is
best-effort so a transient serialize race cannot abort the turn.
step's agent has an auto-generated id (which would break checkpoint resume), backed by the
internal
AIAgent.HasAutoGeneratedIdsignal andWorkflowHostAgent.GetService<Workflow>().What is the impact of these changes?
Additive and off by default. Existing hosted agents are unaffected unless they opt in.
Related Issue
Fixes #
Contribution Checklist