.NET: Store executable function calls bypassed by declaration-only tool calls - #7388
.NET: Store executable function calls bypassed by declaration-only tool calls#7388westey-m wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an opt-in .NET chat-client pipeline decorator to work around FunctionInvokingChatClient terminating early when a declaration-only tool call appears alongside an executable tool call, preventing orphaned backend tool calls (and downstream provider 400s) in mixed backend/frontend tool scenarios.
Changes:
- Introduces
ExecutableFunctionBypassingChatClient, which strips executableFunctionCallContentwhen mixed with declaration-only calls, stores them in session state, and re-injects them next turn as pre-approved tool-approval responses. - Adds
ChatClientAgentOptions.EnableExecutableFunctionBypassing(experimental, defaultfalse) and wires the decorator into the default middleware pipeline when enabled. - Adds a builder extension
UseExecutableFunctionBypassing()plus comprehensive unit tests for non-streaming and streaming behaviors.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| dotnet/src/Microsoft.Agents.AI/ChatClient/ExecutableFunctionBypassingChatClient.cs | New decorator implementing the bypass/store/reinject behavior for mixed executable + declaration-only tool calls (including streaming support). |
| dotnet/src/Microsoft.Agents.AI/ChatClient/ChatClientExtensions.cs | Injects the decorator into the default agent middleware pipeline when enabled. |
| dotnet/src/Microsoft.Agents.AI/ChatClient/ChatClientBuilderExtensions.cs | Adds a public (experimental) builder extension for custom pipelines. |
| dotnet/src/Microsoft.Agents.AI/ChatClient/ChatClientAgentOptions.cs | Adds the opt-in EnableExecutableFunctionBypassing option and clones it in Clone(). |
| dotnet/tests/Microsoft.Agents.AI.UnitTests/ChatClient/ExecutableFunctionBypassingChatClientTests.cs | New unit tests validating bypassing behavior for both GetResponseAsync and GetStreamingResponseAsync. |
| dotnet/tests/Microsoft.Agents.AI.UnitTests/ChatClient/ChatClientAgentOptionsTests.cs | Extends cloning tests to include the new option. |
- Guard enumerator acquisition so pending bypassed calls are restored when the inner client throws synchronously, before the first MoveNextAsync. - Always surface buffered streaming updates, even when stripping empties them, so metadata such as ConversationId and ResponseId is not discarded. - Document that the decorator must sit below ApprovalResponseBindingChatClient, which drops approval responses that have no recorded request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| var allTools = (options?.Tools ?? Enumerable.Empty<AITool>()) | ||
| .Concat(ficc?.AdditionalTools ?? Enumerable.Empty<AITool>()); | ||
|
|
||
| HashSet<string> executable = new(StringComparer.Ordinal); |
There was a problem hiding this comment.
I would change the executable to invocable to better represent the API. InvokeAsync. Not just here but in the docs and introduced Knob/Property.
| HashSet<string> executable = new(StringComparer.Ordinal); | |
| HashSet<string> invocable = new(StringComparer.Ordinal); |
| try | ||
| { | ||
| response = await base.GetResponseAsync(messages, options, cancellationToken).ConfigureAwait(false); | ||
| } | ||
| catch | ||
| { | ||
| RestorePendingBypassedCalls(session, pendingCalls); | ||
| throw; | ||
| } |
There was a problem hiding this comment.
Is this intended to Restore regardless of the Exception or should we be specific to what type of exception are we going to trigger restore?
GitHub Actions
Inactive
| contentLists[i] = tail[i].Contents; | ||
| } | ||
|
|
||
| this.StripAndStoreBypassableExecutableCalls(contentLists, options, session); |
There was a problem hiding this comment.
This strip/store step sits after the try/finally, so it wouldn't run if the consumer abandons the stream early. In that case the buffered tail is discarded and the pending calls pulled out of the state bag at line 128 are lost, which could reintroduce the orphaned-call_id. Should this be in the finally segment?
| /// a no-op: it passes the request through to the inner client unchanged and logs a warning. | ||
| /// </para> | ||
| /// <para> | ||
| /// When the pipeline also contains an <see cref="ApprovalResponseBindingChatClient"/>, this decorator must sit |
There was a problem hiding this comment.
nit: Is there a way to enforce or test for this decorator ordering requirement?
| /// Default is <see langword="false"/>. | ||
| /// </value> | ||
| [Experimental(DiagnosticIds.Experiments.AgentsAIExperiments)] | ||
| public bool EnableExecutableFunctionBypassing { get; set; } |
There was a problem hiding this comment.
nit: Should the name be EnableExecutableFunctionDeferral or something like that? Basically, replacing bypassing with deferral as it seems the function isn't fully bypassed, but executed in the next request.
Motivation & Context
The three participants
The words "backend" and "frontend" are ambiguous here, so this description names the three participants explicitly.
ChatClientAgentThe two kinds of tool
The only difference between them is where the code that actually runs the tool lives.
Tool whose code lives inside the agent process (
AIFunction), for exampleGetOrderStatus. The C# method is compiled into the agent process, so the agent can call it immediately without involving anyone else.Tool that is only declared (
AIFunctionDeclarationthat is not anAIFunction), for exampleHighlightOrderOnScreen. The agent process holds only the name and the parameter schema, no code. The real implementation lives in the browser. The agent process must hand the request back to the calling app, end the turn, and wait for the calling app to send the result on a later request.Both kinds are advertised to the AI service in the same tool list. From the AI service's point of view they are simply two tools, and it has no idea who runs each one.
What goes wrong today
The user types: "What is the status of order 123, and highlight it on my screen".
The AI service answers by requesting both tools at once, each with its own call id:
call_1=GetOrderStatus(code lives in the agent process)call_2=HighlightOrderOnScreen(code lives in the browser)sequenceDiagram participant App as Calling app (browser) participant Agent as Agent process (.NET) participant AI as AI service (Responses API) App->>Agent: "status of order 123 and highlight it" Agent->>AI: message + both tool declarations AI-->>Agent: requests call_1 (GetOrderStatus) and call_2 (HighlightOrderOnScreen) Note over Agent: FICC scans the batch, finds call_2<br/>which it cannot invoke, and abandons<br/>the whole loop. call_1 is returned<br/>unexecuted as well. Agent-->>App: returns call_1 AND call_2 Note over App: the app only knows HighlightOrderOnScreen.<br/>It resolves call_2 and ignores call_1. App->>Agent: result for call_2 only Agent->>AI: history says two calls were requested,<br/>only one result is present AI-->>Agent: 400: call_1 has no matching resultcall_1is left orphaned. The AI service recorded in the history that it requested two calls, so sending the history back with a result for only one of them makes it reject the whole request. This is a rule of the AI service protocol, not a choice the agent makes, and it blocks the common agentic UI scenario where an agent exposes agent process tools and browser tools at the same time.Description & Review Guide
What are the major changes?
A new opt-in, experimental option
ChatClientAgentOptions.EnableExecutableFunctionBypassing(default
false) and a new internal decoratorExecutableFunctionBypassingChatClientthat sitsdirectly above
FunctionInvokingChatClient(FICC) in the pipeline.The decorator works in two directions. On the way out, when a response contains both an
invocable call and a declaration only call, it removes the invocable calls from the response and
stores them in the session's
AgentSessionStateBag, returning only the declaration only calls tothe calling app. On the way back in, it takes the stored calls out of the session and re injects
them as pre approved
ToolApprovalResponseContent, which FICC reconstructs and executes, producingthe missing
FunctionResultContent.sequenceDiagram participant App as Calling app (browser) participant EFB as ExecutableFunctionBypassingChatClient participant FICC as FICC participant AI as AI service App->>EFB: "status of order 123 and highlight it" EFB->>FICC: passes through FICC->>AI: message + both tool declarations AI-->>FICC: requests call_1 and call_2 FICC-->>EFB: returns call_1 and call_2 unexecuted Note over EFB: sees both kinds in one response.<br/>REMOVES call_1 and STORES it<br/>in the session state bag. EFB-->>App: returns ONLY call_2 Note over App: resolves HighlightOrderOnScreen App->>EFB: result for call_2 Note over EFB: pulls call_1 back from the session<br/>and injects it as a pre approved<br/>ToolApprovalResponseContent EFB->>FICC: message + approved call_1 Note over FICC: recognises the approval and<br/>actually executes GetOrderStatus FICC->>AI: consistent history, every call has a result AI-->>FICC: final text response FICC-->>EFB: response EFB-->>App: final responseThe reason this works is that the AI service never sees
call_1without a result. Because thedecorator strips
call_1from the response before it is written to history, that turn requestedexactly one tool and received exactly one result.
GetOrderStatusis not lost, only deferred by oneturn, after which it runs for real and the model gets its result.
Approval responses are the vehicle because FICC only re executes calls that arrive from incoming
history as approval responses; it ignores bare
FunctionCallContentin history. This mirrors theexisting
ApprovalNotRequiredFunctionBypassingChatClient, which uses the same state bag plussynthetic approval mechanism. A
UseExecutableFunctionBypassingbuilder extension is included forcustom pipelines.
What is the impact of these changes?
None by default. The option is opt in and defaults to
false, so the decorator is not added tothe pipeline unless explicitly enabled. When it is enabled, mixed turns complete correctly instead
of failing with a provider 400.
The bypass is deliberately gated on both kinds of call being present in the same response. A
response containing only invocable calls is already handled by FICC and never surfaces unexecuted,
and a response containing only declaration only calls is the normal browser tool flow that must
pass through untouched. Other FICC termination causes (max iterations, unknown or unresolvable
calls) also surface unexecuted invocable calls, but there the caller legitimately should receive
them, so bypassing must not trigger. The decorator is also a no op pass through when invoked
without an agent run context.
Ordering matters: the decorator must sit below
ApprovalResponseBindingChatClient, which dropsany approval response that has no request it recorded so that a forged approval cannot execute. The
approvals injected here are synthetic and have no such request, so the wrong order would silently
discard them. The default agent pipeline already orders them correctly.
If a request fails, the pending calls are restored to the session so that a transient error does
not silently drop a call that was never executed.
What do you want reviewers to focus on?
The streaming path in
GetStreamingResponseAsync. It intentionally does not buffer the wholeresponse: updates stream live until a surfaced call appears, then the tail is held only for as
long as it could still be stripped. FICC flips
FunctionCallContent.InformationalOnlytotruein place when it invokes a call, so a buffered tail whose calls have all become informational has
nothing left to bypass and is released, resuming live streaming. Only a genuinely bypassable batch,
where FICC terminated the loop and left the calls non informational, is held to end of stream.
Related Issue
Fixes #6922
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.