Workflow Evaluation

Comprehensive guide to evaluating multi-agent workflows with AgentEval and Microsoft Agent Framework (MAF)

AgentEval provides first-class support for evaluating Microsoft Agent Framework (MAF) workflows. Build real workflow pipelines with WorkflowBuilder, then evaluate them with structured assertions, timing validation, and rich execution reporting.

Overview

MAF workflows orchestrate multiple agents in sequential or complex execution patterns:

• Sequential pipelines: One agent's output feeds the next (Planner → Researcher → Writer → Editor)
• Tool-enabled workflows: Agents with function calling working together
• Event streaming: Real-time execution monitoring with AgentResponseUpdateEvent
• Graph extraction: Automatic workflow structure analysis

AgentEval captures the complete workflow execution, enabling you to:

• Evaluate execution order: Assert agents executed in correct sequence
• Validate timing: Ensure workflows complete within acceptable timeframes
• Track tool usage: Monitor function calls across all agents
• Generate visualizations: Export Mermaid diagrams and timeline JSON

Quick Start

using AgentEval.MAF;
using AgentEval.Workflows;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.WorkflowBuilder;

// 1. Create MAF workflow with WorkflowBuilder
var chatClient = new AzureOpenAIClient(endpoint, credential)
    .GetChatClient(deployment).AsIChatClient();

var planner = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "Planner", 
    ChatOptions = new() { Instructions = "Create content plans" }
});
var writer = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "Writer", 
    ChatOptions = new() { Instructions = "Write content from plans" }
});

// Bind agents as executors with event emission
var plannerBinding = planner.BindAsExecutor(emitEvents: true);
var writerBinding = writer.BindAsExecutor(emitEvents: true);

// Build MAF workflow
var workflow = new WorkflowBuilder(plannerBinding)
    .AddEdge(plannerBinding, writerBinding)
    .Build();

// 2. Create AgentEval adapter
var executorIds = new[] { "Planner", "Writer" };
var adapter = MAFWorkflowAdapter.FromMAFWorkflow(
    workflow, "ContentPipeline", executorIds, "PromptChaining");

// 3. Evaluate with WorkflowEvaluationHarness
var harness = new WorkflowEvaluationHarness(verbose: true);
var testCase = new WorkflowTestCase
{
    Name = "Content Generation Test",
    Input = "Write an article about AI agents",
    ExpectedSteps = ["Planner", "Writer"]
};

var result = await harness.RunWorkflowTestAsync(adapter, testCase);

// 4. Assert on execution
result.ExecutionResult!.Should()
    .HaveStepCount(2)
    .HaveExecutedInOrder("Planner", "Writer")
    .HaveCompletedWithin(TimeSpan.FromMinutes(2))
    .HaveNoErrors();

Core Concepts

MAF Workflow Integration

AgentEval integrates with Microsoft Agent Framework (MAF)'s native workflow system:

1. Agent Binding: Agents are bound as MAF executors with event emission enabled
2. WorkflowBuilder: Use MAF's WorkflowBuilder to define execution graphs
3. Event Streaming: MAF emits real-time execution events via WatchStreamAsync()
4. Adapter Pattern: MAFWorkflowAdapter bridges MAF workflows to AgentEval evaluation

Workflow Test Result

The WorkflowTestResult captures the complete evaluation:

public record WorkflowTestResult
{
    // Test metadata
    public required string TestName { get; init; }
    public required TimeSpan Duration { get; init; }
    public required bool Passed { get; init; }
    
    // Execution details
    public WorkflowExecutionResult? ExecutionResult { get; init; }
    
    // Error information
    public string? ErrorMessage { get; init; }
    public Exception? Exception { get; init; }
}

public record WorkflowExecutionResult  
{
    // Final workflow output
    public required string FinalOutput { get; init; }
    
    // Step-by-step execution
    public required IReadOnlyList<ExecutorStep> Steps { get; init; }
    
    // Performance metrics
    public required PerformanceMetrics Performance { get; init; }
    
    // Tool usage across all agents
    public ToolUsageResult? ToolUsage { get; init; }
    
    // Graph structure (extracted from MAF)
    public WorkflowGraphDefinition? GraphDefinition { get; init; }
    
    // Rich execution timeline
    public WorkflowTimelineResult Timeline { get; init; }
}

Executor Step Results

Each agent (executor) execution is captured as an ExecutorStep:

public record ExecutorStep
{
    // Agent identification
    public required string ExecutorId { get; init; }     // "Planner", "Writer", etc.
    public required int StepIndex { get; init; }          // 0, 1, 2...
    
    // Output and timing
    public required string Output { get; init; }         // Agent's response
    public required TimeSpan StartOffset { get; init; }  // When MAF started running the step (from workflow start)
    public required TimeSpan Duration { get; init; }     // How long MAF ran it (see "Step timing" below)
    
    // Tool usage (if agent used tools)
    public IReadOnlyList<ToolCallRecord>? ToolCalls { get; init; }
    public bool HasToolCalls => ToolCalls?.Count > 0;
    
    // Error information
    public IReadOnlyList<AgentError>? Errors { get; init; }
    public bool HasErrors => Errors?.Count > 0;
    
    // Performance metrics
    public int? InputTokens { get; init; }
    public int? OutputTokens { get; init; }
    public decimal? EstimatedCost { get; init; }
}

MAF Event Processing

AgentEval captures these MAF workflow events:

MAF Event	Purpose
SuperStepStartedEvent	Workflow superstep begins
ExecutorInvokedEvent	A message is delivered to an executor
AgentResponseUpdateEvent	Streaming token from LLM
ExecutorCompletedEvent	The executor finished handling that message
SuperStepCompletedEvent	Workflow superstep ends

Events are processed by MAFWorkflowEventBridge into AgentEval's evaluation model.

MAF raises ExecutorInvokedEvent once per delivered message. The bridge groups a chat executor's incoming messages with the turn that processes them, and records a second turn separately even within the same superstep. Function deliveries to one executor in a superstep form one activation. Each parallel executor owns its own text, tool calls, response usage and timing. Completed activations are displayed in the executorIds order supplied to FromMAFWorkflow, with ordinal name order for any remaining executors; actual timestamps retain the execution order and overlap.

Step timing. Use a disposable MAFWorkflowInstrumentation owner to build the root and each declared child workflow. Keep it alive until all evaluations finish. It gives each built workflow a dedicated telemetry source; the bridge also assigns each evaluation its own MAF session identity. A plain .WithOpenTelemetry() enables observability but does not establish enough ownership for AgentEval to certify executor timing. The existing adapter entry points still execute those workflows and retain output and structural evidence, with an explicit timing-unavailable reason.

using var timing = new MAFWorkflowInstrumentation();
var workflow = timing.Build(new WorkflowBuilder(start).WithOutputFrom(start));
var agent = MAFWorkflowAdapter.FromMAFWorkflow(workflow, "Example", ["Start"]);
var result = await agent.ExecuteWorkflowAsync("input");

Build explicitly replaces that builder's telemetry configuration; sensitive payload capture stays disabled. It builds once, preserving the workflow's normal state semantics. The bridge captures monotonic producer start/end timestamps. A step spans its first delivery through its final completion, including waits between accumulated chat input and the turn token, or a request and its response. Declared nested hosts include their child intervals. Parallel siblings retain their own intervals regardless of display order or event-queue delays.

Execution boundary and partial evidence. Work triggered by the evaluated workflow belongs to that execution, including external and nested operations. Membership does not make a child's duration interchangeable with its parent's. The parent's elapsed time includes awaited child work; parallel durations need not sum to whole-run elapsed time. Whole-workflow elapsed time remains measured even if individual step or tool timing is unavailable. Retain observed output, call/result identity and known completion independently of timing. Missing or ambiguous timing is not zero and cannot satisfy a duration assertion; both workflow tool-assertion builders fail validation with an explanation. The general tool-usage assertion API retains its existing inconclusive behavior. Queued event arrival timestamps alone are not producer execution durations.

The adapter preserves the caller's telemetry configuration. Unregistered telemetry sources, missing producer evidence or an unfinished span produce TimingNotMeasuredReason; executor budgets cannot pass on that missing evidence. MAF's convenience agent-workflow builders return workflows without these spans enabled, so their executor timing is unavailable. Event arrival timestamps cannot establish executor runtime: the producer can finish while its consumer is paused. Custom AgentEval event sources retain their existing explicit or arrival timing contract.

Parallel steps are timed to each executor's own completion. Recorded fan-out branches carry their actual executor IDs and timing; siblings do not create an edge between each other merely because their events arrived consecutively. HaveParallelExecution requires at least two distinct branch IDs on recorded steps; HaveParallelBranchCount counts those IDs, including after trace replay. Declaring a fan-out or selecting only one target does not satisfy the parallel-execution assertion. Incoming edges use reflected topology and observed deliveries. At a join, matching message IDs or an unambiguous return value identify the sender. Collection payloads retain their whole-message identity as well as member matching, so older array-valued barrier inputs keep their source activation. If equal payloads could come from several conditional sources, the bridge leaves the incoming traversal unrecorded rather than guessing which condition passed. A senderless turn token also cannot identify one of several incoming edges.

MAF's fan-in barrier releases individual messages; it does not aggregate them into an array. With agent chat protocols, it can leave message accumulation waiting for another token. The recorder preserves each observed turn and any remaining accumulation separately; the latter does not erase the last produced output. Use ordinary incoming edges when each upstream agent should trigger its own turn. The barrier's message-routing behavior is controlled by MAF.

An unequal-depth barrier can retain source messages across supersteps, including messages from several activations of the same source. Matching payload evidence keeps every identifiable contributing activation attached to the eventual join; consumed evidence is not reused for a later barrier cycle. Repeated turns or trailing accumulation on one selected fan-out target retain that target's branch identity for the same source activation.

A step whose invocation, completion or agent turn was not observed carries TimingNotMeasuredReason. ForExecutor(...).HaveCompletedWithin(...) fails with "time not measured" and the reason, regardless of the numeric duration placeholder. Measured zero durations remain valid. Trace recording/replay and JSON exports retain the reason; per-executor trace totals omit any executor with unmeasured activations. Custom sources without an explicit missing-timing reason keep arrival timing.

Tool evidence and replay. A requested tool is retained even when no result arrives. Only an observed result sets WasExecuted, including a legitimate null result. Workflow trace JSON retains that execution marker and the call ID; replay preserves unknown completion timing. Legacy traces without the marker keep execution unconfirmed. Workflow JSON exports also include WasExecuted. The queued MAF bridge sets ExecutorToolCallEvent.HasTiming to false even for paired results: executor spans do not measure individual tools. With timing unavailable, general ToolUsage.Should() duration assertions are inconclusive, while workflow-wide and executor-scoped tool-duration validation fails; exports omit tool durations. Custom sources with actual tool-boundary measurements retain HasTiming = true (the compatibility default).

Timeline invocations carry OutcomeNotMeasuredReason and TimingNotMeasuredReason. Unknown outcomes are excluded from success/failure counts; unknown durations are excluded from timing statistics. Numeric duration aggregates summarize only the measured subset, so check UnmeasuredDurationCount before treating them as complete. The text timeline and Microsoft logging name unavailable values and their reasons. When a timeline supplies a TestResult's evaluation input, an explicit unknown outcome retains the __tool_not_executed__: marker and reason rather than becoming a tool error. Upgrade: ToolStatistics.SuccessRate is now double?, calculated over observed outcomes; it is null when every recorded outcome is unmeasured. Check for null when formatting or comparing it. Existing directly constructed timeline entries without missing-measurement markers retain their prior interpretation.

Workflow trace JSON stores observed traversals separately from declared topology, including edge IDs, source/target activation indices, routing evidence and exact traversal time. Recording and replay preserve traversal assertions through loops. Old traces without observed traversals leave that collection absent; legacy step edges without activation indices replay with index -1, rather than inventing an association with step zero.

Agent-builder recording contracts

The offline test suite executes the real MAF builders using scripted chat clients. Orchestration executors remain visible as steps:

Builder	Recorded infrastructure	Agent-turn contract
Sequential	OutputMessages	One turn per agent in chain order; terminal output contains the produced messages.
Concurrent	Start, Batcher/<agent>, ConcurrentEnd	Each selected agent owns its turn and branch; batchers feed the fan-in. Display order follows configured names, while terminal message order may follow arrival order.
Handoff	HandoffStart, HandoffEnd	The actual handoff tool request selects the next agent. MAF exposes selector edges as fan-outs in reflected topology; one selected successor does not count as parallel execution. Tool-only messages can contribute empty lines to rendered terminal output.
Group chat	GroupChatHost between participant turns	The manager chooses the speaker and termination. Actual agent turns stay separate, including a participant speaking again.

MAF registers GroupChatHost through a factory typed as the base Executor, hiding its chat-protocol type from static binding reflection. Its message and turn-token deliveries may therefore appear as adjacent orchestration activations when scheduling puts them in separate supersteps. Its recorded duration describes those deliveries, not the whole group conversation; use participant step budgets for agent work and the workflow timeout for the conversation. The tests allow this infrastructure scheduling variation while checking exact participant turns, outputs, observed edges, edge-to-step indexes and measured agent timing.

The shape suite also covers direct function/agent chains and loops, fan-out, ordinary and barrier fan-in, nested workflows and request/response ports. The explicit routing and nested-host limits described here are tested; unsupported or unmeasured behavior is not certified as a successful measurement.

Nested workflows and external answers

Nested executor steps use paths such as Sub/A and Sub/B. Supply those paths in executorIds to choose display names and ordering. The host starts with its input and runs through its observed children; its queueing call alone is not its duration. Nested results are buffered until the stream finishes so a host can precede its children while retaining the full interval. Incomplete children or an interrupted stream leave the host's timing not measured. Nested agent inputs receive the chat protocol's turn token, and forwarded agent events are attributed using their agent ID so child text, tool calls and usage stay with the child.

MAF flattens child events without their parent scope. Raw executor IDs must therefore be unique across the workflow tree; duplicate IDs are rejected before execution. Nested agent output must identify one bound agent within its host. MAF also cannot forward input through a host whose entry is directly another catch-all host; add an entry executor with an explicit input type inside that workflow. Multi-level nesting with concrete entry executors is supported.

Use the responder overload to answer a request during execution:

var adapter = MAFWorkflowAdapter.FromMAFWorkflow(
    workflow, "approval workflow", ["Approval", "Processor"], workflowType: null,
    requestResponder: (request, cancellationToken) =>
        ValueTask.FromResult<ExternalResponse?>(request.CreateResponse("approved")));

The callback receives the original request and run cancellation token. Return null to decline; a declined, failed or mismatched response fails the run with the request and port identity. The original four-argument overload remains available. The answer completes the existing port activation, including time spent waiting, and its payload becomes the port's output; it does not create a self-loop. Subsequent genuine requests are separate activations.

For a request inside a subworkflow, MAF forwards an ExternalRequest message from its host. Configure an outer request port to receive that wrapped message and route its ExternalResponse back to the host. Use type-filtering predicates (request => request is not null, response => response is not null) so normal host output is not routed as another request. The callback sees the outer port's request. A nested request with no relay fails promptly with the child port name and setup guidance.

Timeout Handling

An executor yielding a workflow output does not complete the run. AgentEval continues recording until the MAF event stream ends, preserving later steps, repeated yields and errors. For MAF's sequential and concurrent agent builders, chat-list output is rendered as message text separated by newlines; the original input message is excluded by identity, so a new reply with identical text is retained.

If the event stream ends on cancellation, the adapter propagates cancellation. The harness reports its own timeout as a failed test and propagates caller cancellation. An unanswered request port returns a workflow error naming its port and request ID; the request envelope is not an executor answer.

MAF workflows may not honor cancellation tokens during active LLM calls. AgentEval implements a hard timeout pattern in critical samples:

var workflowTask = harness.RunWorkflowTestAsync(adapter, testCase, options);
var hardTimeout = Task.Delay(TimeSpan.FromMinutes(5));

if (await Task.WhenAny(workflowTask, hardTimeout) == hardTimeout)
{
    Console.WriteLine("⏱️ Workflow exceeded hard timeout — moving on.");
    return; // Graceful timeout handling
}

var result = await workflowTask;

This prevents indefinite hangs in CI/CD environments where workflows might stall.

Graph Structure Extraction

AgentEval automatically extracts workflow graph structure from MAF workflows:

public record WorkflowGraphDefinition
{
    public IReadOnlyList<WorkflowNode> Nodes { get; init; }
    public IReadOnlyList<WorkflowEdge> Edges { get; init; }
    public string? EntryNodeId { get; init; }
    public IReadOnlyList<string> ExitNodeIds { get; init; }
}

public record WorkflowNode
{
    public required string NodeId { get; init; }         // "Planner", "Writer"
    public string? DisplayName { get; init; }            // Human-readable name
    public bool IsEntryPoint { get; init; }             // First node
    public bool IsExitPoint { get; init; }              // Last node
}

public record WorkflowEdge  
{
    public required string EdgeId { get; init; }
    public required string SourceNodeId { get; init; }   // "Planner"
    public required string TargetNodeId { get; init; }   // "Writer"
    public EdgeType EdgeType { get; init; }              // Sequential, etc.
}

Creating MAF Workflows

Sequential Pipeline Pattern

The most common workflow pattern is a sequential pipeline where each agent builds on the previous agent's work:

// 1. Create AI agents with distinct roles
var azureClient = new AzureOpenAIClient(endpoint, credential);
var chatClient = azureClient.GetChatClient(deployment).AsIChatClient();

var planner = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "Planner",
    Description = "Plans content structure",
    ChatOptions = new ChatOptions
    {
        Instructions = """
            You are a content planning specialist. Create structured plans with:
            1. Logical outline with main sections
            2. Key research points per section  
            3. Target audience and tone
            4. Suggested word count
            """
    }
});

var researcher = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "Researcher", 
    Description = "Researches topics based on a plan",
    ChatOptions = new ChatOptions
    {
        Instructions = """
            You are a research specialist. Given a content plan:
            1. Identify research needs for each section
            2. Synthesize information into organized research notes
            3. Include key facts, data points, and expert insights
            4. Note current trends and credible sources
            """
    }
});

var writer = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "Writer",
    Description = "Writes comprehensive articles from research", 
    ChatOptions = new ChatOptions
    {
        Instructions = """
            You are an experienced technical writer. Transform research into:
            1. Well-structured, flowing prose
            2. Clear, accessible language with technical accuracy
            3. Practical examples and actionable insights
            4. Engaging introduction and strong conclusion
            """
    }
});

var editor = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "Editor",
    Description = "Polishes and refines articles",
    ChatOptions = new ChatOptions
    {
        Instructions = """
            You are a professional editor. Polish the draft for:
            1. Clarity, flow, and engagement
            2. Improved sentence structure and word choice
            3. Consistent tone and style
            4. Grammar, punctuation, and formatting
            """
    }
});

// 2. Bind agents as MAF executors with event emission
var plannerBinding = planner.BindAsExecutor(emitEvents: true);
var researcherBinding = researcher.BindAsExecutor(emitEvents: true);
var writerBinding = writer.BindAsExecutor(emitEvents: true);
var editorBinding = editor.BindAsExecutor(emitEvents: true);

// 3. Build the workflow graph with WorkflowBuilder
var workflow = new WorkflowBuilder(plannerBinding)    // Start with planner
    .AddEdge(plannerBinding, researcherBinding)        // Planner → Researcher
    .AddEdge(researcherBinding, writerBinding)         // Researcher → Writer  
    .AddEdge(writerBinding, editorBinding)             // Writer → Editor
    .Build();

// 4. Create AgentEval adapter
var executorIds = new[] { "Planner", "Researcher", "Writer", "Editor" };
var adapter = MAFWorkflowAdapter.FromMAFWorkflow(
    workflow, "ContentPipeline", executorIds, "PromptChaining");

Tool-Enabled Workflow Pattern

Agents can use tools (function calling) within workflows. Each agent's tools are tracked both individually and at the workflow level:

// Create tools for the TripPlanner agent
static string GetInfoAbout([Description("City to get info about")] string city)
{
    return $"Information about {city}: Beautiful city with rich history...";
}

static string SearchFlights([Description("Origin city")] string from, 
                           [Description("Destination city")] string to)
{
    return $"Found flights from {from} to {to}: Flight AA123 at 10:30 AM...";
}

static string BookFlight([Description("Flight number")] string flightNumber)
{
    return $"Booked flight {flightNumber}. Confirmation: ABC123";
}

static string BookHotel([Description("City for hotel")] string city)
{
    return $"Booked hotel in {city}. Confirmation: HTL456";
}

// Create agents with tools configured
var tripPlanner = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "TripPlanner",
    Description = "Gathers city information and plans trip itinerary",
    ChatOptions = new ChatOptions
    {
        Instructions = "Use GetInfoAbout tool for EACH city mentioned. Create day-by-day itinerary.",
        Tools = [AIFunctionFactory.Create(GetInfoAbout)]
    }
});

var flightAgent = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "FlightReservation", 
    Description = "Searches and books flights between cities",
    ChatOptions = new ChatOptions
    {
        Instructions = "Use SearchFlights first, then BookFlight for each journey leg.",
        Tools = [
            AIFunctionFactory.Create(SearchFlights),
            AIFunctionFactory.Create(BookFlight)
        ]
    }
});

var hotelAgent = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "HotelReservation",
    Description = "Books hotels for each city in the trip", 
    ChatOptions = new ChatOptions
    {
        Instructions = "Use BookHotel for EACH city that needs accommodation.",
        Tools = [AIFunctionFactory.Create(BookHotel)]
    }
});

var presenter = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "Presenter",
    Description = "Creates final trip presentation",
    ChatOptions = new() { Instructions = "Summarize the complete trip with all bookings and confirmations." }
});

// Build tool-enabled workflow
var tripPlannerBinding = tripPlanner.BindAsExecutor(emitEvents: true);
var flightBinding = flightAgent.BindAsExecutor(emitEvents: true);
var hotelBinding = hotelAgent.BindAsExecutor(emitEvents: true);
var presenterBinding = presenter.BindAsExecutor(emitEvents: true);

var toolWorkflow = new WorkflowBuilder(tripPlannerBinding)
    .AddEdge(tripPlannerBinding, flightBinding)
    .AddEdge(flightBinding, hotelBinding)
    .AddEdge(hotelBinding, presenterBinding)
    .Build();

var toolAdapter = MAFWorkflowAdapter.FromMAFWorkflow(
    toolWorkflow, "TripPlanner", 
    ["TripPlanner", "FlightReservation", "HotelReservation", "Presenter"]);

Chat-boundary capture inside workflows (Glass Box)

WorkflowChatRecording (AgentEval.MAF.Tracing) pre-wires each executor's IChatClient with UseTraceRecording and collects one AgentTrace per executor:

var recording = new WorkflowChatRecording(SamplePreset.AuditGrade);
var plannerClient = recording.Instrument("TripPlanner", rawPlannerModel);   // wrap before building the agent
// … build each executor's agent from recording.Instrument(id, rawModel), then BindAsExecutor …
// after the run:
var plannerTrace = recording.TraceFor("TripPlanner");

⚠️ Known limitation (validated). When an instrumented agent is invoked directly (RunAsync/RunStreamingAsync) the recorder captures the full per-turn chat boundary. But when that agent is BindAsExecutor-ed into a MAF Workflow and run via InProcessExecution, the executor consumes the agent's result internally and does not route the response back through the instrumented client — so the per-executor trace sees the request but not the response (no per-turn tokens/finish/tool-calls inside the workflow). This is a MAF executor-boundary behaviour, not a Glass Box bug.

What is reachable inside a workflow today (no MAF change): the per-executor cost ledger — tokens + finish reason — is already extracted from MAF's public WorkflowEvent stream by MAFWorkflowEventBridge into an ExecutorAgentResponseEvent; wiring that into a workflow-level reconciliation gives per-agent cost attribution without touching MAF. Full per-turn chat-boundary parity inside a live workflow requires an upstream MAF extension point (a per-executor trace hook).

For a directly-invoked agent, each AgentTrace reconciles with its agent-boundary trace via the Trace Fidelity family, and the per-turn detail surfaces in Mission Control's trace waterfall.

Workflow Evaluation Patterns

Basic Sequential Workflow Evaluation

[Fact]
public async Task ContentPipeline_Should_ExecuteInCorrectOrder()
{
    // Arrange
    var (workflow, executorIds) = CreateContentPipelineWorkflow();
    var adapter = MAFWorkflowAdapter.FromMAFWorkflow(
        workflow, "ContentPipeline", executorIds, "PromptChaining");
        
    var harness = new WorkflowEvaluationHarness(verbose: true);
    var testCase = new WorkflowTestCase
    {
        Name = "Content Generation Pipeline — AI Testing Article",
        Input = "Write a comprehensive article about AI agent evaluation testing",
        ExpectedSteps = ["Planner", "Researcher", "Writer", "Editor"]
    };
    
    var options = new WorkflowTestOptions
    {
        Timeout = TimeSpan.FromMinutes(5),
        Verbose = true
    };

    // Act - with timeout handling
    var workflowTask = harness.RunWorkflowTestAsync(adapter, testCase, options);
    var hardTimeout = Task.Delay(TimeSpan.FromMinutes(5));
    
    if (await Task.WhenAny(workflowTask, hardTimeout) == hardTimeout)
    {
        Assert.True(false, "Workflow exceeded hard timeout");
    }
    
    var result = await workflowTask;

    // Assert
    result.ExecutionResult!.Should()
        .HaveStepCount(4, because: "pipeline has 4 distinct stages")
        .HaveExecutedInOrder("Planner", "Researcher", "Writer", "Editor")
        .HaveCompletedWithin(TimeSpan.FromMinutes(3), because: "reasonable time for content generation")
        .HaveNoErrors(because: "clean execution is required")
        .HaveNonEmptyOutput()
        .Validate();
}

Per-Executor Evaluation

Validate individual agent performance within the workflow:

// Per-executor detailed assertions
result.ExecutionResult!.Should()
    .ForExecutor("Planner")
        .HaveNonEmptyOutput()
        .HaveCompletedWithin(TimeSpan.FromSeconds(60), because: "planning should be reasonably fast")
        .And()
    .ForExecutor("Researcher")
        .HaveNonEmptyOutput()
        .HaveDurationGreaterThan(TimeSpan.FromSeconds(5), because: "research takes time")
        .And()
    .ForExecutor("Writer")
        .HaveNonEmptyOutput()
        .And()
    .ForExecutor("Editor")
        .HaveNonEmptyOutput()
        .And()
    .Validate();

Graph Structure Validation

Verify the workflow graph was correctly extracted and executed:

// Graph structure assertions
result.ExecutionResult!.Should()
    .HaveGraphStructure()
    .HaveNodes("Planner", "Researcher", "Writer", "Editor")
    .HaveEntryPoint("Planner", because: "planning is the starting point")
    .HaveTraversedEdge("Researcher", "Writer")
    .HaveUsedEdgeType(EdgeType.Sequential)
    .HaveExecutionPath("Planner", "Researcher", "Writer", "Editor")
    .Validate();

Tool Usage Evaluation

For workflows with tool-enabled agents, validate tool call patterns:

[Fact]
public async Task TripPlannerWorkflow_Should_UseToolsCorrectly()
{
    // Arrange - tool-enabled workflow
    var (workflow, executorIds) = CreateTripPlannerWorkflow();
    var adapter = MAFWorkflowAdapter.FromMAFWorkflow(
        workflow, "TripPlanner", executorIds);
        
    var result = await harness.RunWorkflowTestAsync(adapter, testCase);

    // Assert tool usage at workflow level
    if (result.ExecutionResult!.ToolUsage != null)
    {
        result.ExecutionResult.Should()
            .HaveCalledTool("GetInfoAbout", because: "TripPlanner must research cities")
                .WithoutError()
            .And()
            .HaveCalledTool("SearchFlights")
                .BeforeTool("BookFlight", because: "can't book without search results")
                .WithoutError() 
            .And()
            .HaveCalledTool("BookFlight")
                .WithoutError()
            .And()
            .HaveCalledTool("BookHotel", because: "must book hotels")
                .WithoutError()
            .And()
            .HaveNoToolErrors(because: "all tools must succeed for quality output")
            .HaveAtLeastTotalToolCalls(4, because: "workflow uses at least 4 tool calls")
            .Validate();
    }
}

Workflow Assertions API

Basic Workflow Structure

result.ExecutionResult!.Should()
    .HaveStepCount(4)                                    // Exact number of steps
    .HaveAtLeastSteps(3)                                // Minimum steps
    .HaveExecutedInOrder("A", "B", "C")                 // Sequential execution
    .HaveCompletedWithin(TimeSpan.FromMinutes(5))       // Time constraint
    .HaveNoErrors();                                     // Error-free execution

Individual Executor Validation

result.ExecutionResult!.Should()
    .ForExecutor("Planner")
        .HaveNonEmptyOutput()                           // Has output
        .HaveOutputContaining("plan")                   // Output content check
        .HaveCompletedWithin(TimeSpan.FromSeconds(30))  // Individual timing
        .And()
    .ForExecutor("Writer")
        .HaveNonEmptyOutput()
        .HaveToolCalls()                                // Used tools (if expected)
        .And();

Graph Structure Assertions

result.ExecutionResult!.Should()
    .HaveGraphStructure()                               // Graph was extracted
    .HaveNodes("A", "B", "C", "D")                      // Expected nodes
    .HaveEntryPoint("A")                               // Entry node
    .HaveExitPoint("D")                                // Exit node  
    .HaveTraversedEdge("A", "B")                       // Specific edge used
    .NotHaveTraversedEdge("A", "C")                    // Edge not used 
    .HaveExecutionPath("A", "B", "C", "D");            // Complete path

Tool Usage Validation

// At workflow level (aggregated across all agents)
result.ExecutionResult!.Should()
    .HaveCalledTool("SearchFlights")                    // Tool was called
        .AtLeast(1.Times())                            // Call frequency
        .WithoutError()                                // No tool errors
        .WithArgument("from", "Seattle")               // Specific argument
        .BeforeTool("BookFlight")                       // Call ordering
        .And()
    .HaveToolCallPattern("Search*", "Book*")           // Pattern matching
    .HaveNoToolErrors()                                // Global tool success
    .HaveAtLeastTotalToolCalls(3);                     // Minimum tool usage

Performance and Cost Validation

result.ExecutionResult!.Performance!.Should()
    .HaveTotalDurationUnder(TimeSpan.FromMinutes(5))   // Overall timing
    .HaveEstimatedCostUnder(0.25m)                     // Overall cost 
    .HaveTokenUsageUnder(5000)                         // Token limits
    .HaveNoTimeouts();                                  // No timeout errors

// Individual step performance
result.ExecutionResult!.Should()
    .ForExecutor("SlowStep")
        .HaveCompletedWithin(TimeSpan.FromSeconds(30));

Complex Assertion Chaining

Combine multiple assertion types for comprehensive validation:

result.ExecutionResult!.Should()
    // Basic structure
    .HaveStepCount(4)
    .HaveExecutedInOrder("Planner", "Researcher", "Writer", "Editor")
    .HaveCompletedWithin(TimeSpan.FromMinutes(3))
    .HaveNoErrors()
    
    // Per-executor validation
    .ForExecutor("Planner")
        .HaveNonEmptyOutput()
        .HaveCompletedWithin(TimeSpan.FromSeconds(60))
        .And()
    .ForExecutor("Writer")
        .HaveNonEmptyOutput()
        .And()
        
    // Graph validation
    .HaveGraphStructure()
    .HaveEntryPoint("Planner")
    .HaveExecutionPath("Planner", "Researcher", "Writer", "Editor")
    
    // Tool validation (if applicable)
    .HaveNoToolErrors()
    
    // Performance validation
    .Performance!.Should()
        .HaveEstimatedCostUnder(0.20m)
        .And()
    
    // Final validation
    .Validate();

Visualization and Export

Mermaid Diagram Generation

Generate flowchart diagrams for workflow visualization:

using AgentEval.Workflows.Serialization;

// Execute workflow
var result = await harness.RunWorkflowTestAsync(adapter, testCase);

// Generate Mermaid diagram
var mermaid = WorkflowSerializer.ToMermaid(result.ExecutionResult!);
Console.WriteLine(mermaid);

// Output:
// ```mermaid
// graph TD
//     Planner([Planner])
//     Researcher[Researcher]
//     Writer[Writer]
//     Editor[[Editor]]
//     
//     Planner --> Researcher
//     Researcher --> Writer
//     Writer --> Editor
//     
//     classDef executed fill:#90EE90,stroke:#228B22
//     class Planner,Researcher,Writer,Editor executed
// ```

// Save to file for external viewing
var mermaidPath = Path.GetTempFileName() + ".mmd";
File.WriteAllText(mermaidPath, mermaid);
Console.WriteLine($"Mermaid diagram saved to: {mermaidPath}");
Console.WriteLine($"View at: https://mermaid.live");

Timeline JSON Export

Export detailed execution timeline for analysis tools:

// Generate timeline JSON with full execution details
var timeline = WorkflowSerializer.ToTimelineJson(result.ExecutionResult!);
Console.WriteLine($"Timeline JSON: {timeline.Length} characters");

// Timeline includes:
// - Step-by-step execution with timestamps
// - Tool call details and durations
// - Token usage and cost estimates
// - Error information (if any)
// - Performance metrics

// Export for external analysis
var timelineData = JsonSerializer.Deserialize<WorkflowTimelineData>(timeline);
foreach (var step in timelineData.Steps)
{
    Console.WriteLine($"[{step.StartOffset}] {step.ExecutorId}: {step.Duration}ms");
    if (step.ToolCalls?.Count > 0)
    {
        foreach (var tool in step.ToolCalls) 
        {
            Console.WriteLine($"  🔧 {tool.Name} ({tool.Duration}ms)");
        }
    }
}

ASCII Timeline Visualization

For console/CI environments, generate ASCII timeline diagrams:

// Generate ASCII timeline diagram
var asciiDiagram = result.ExecutionResult!.Timeline.ToAsciiDiagram(80);
Console.WriteLine(asciiDiagram);

// Output:
// Timeline (66.8s total):
// ├─ Planner     ████████████                          (20.1s)
// ├─ Researcher  ──────────────██████                  (14.1s)  
// ├─ Writer      ─────────────────────████████████     (17.5s)
// └─ Editor      ──────────────────────────────────    (0.0s)
//    0s    10s   20s   30s   40s   50s   60s   70s

// Include tool usage if present
if (result.ExecutionResult.ToolUsage != null)
{
    Console.WriteLine($"Tool calls: {result.ExecutionResult.ToolUsage.Count} total");
    Console.WriteLine($"Tool time: {result.ExecutionResult.Timeline.TotalToolTime.TotalMilliseconds:F0}ms");
    Console.WriteLine($"Tool efficiency: {result.ExecutionResult.Timeline.ToolTimePercentage:F1}%");
}

Best Practices

1. Use Clear Agent Names and Instructions

// Good - descriptive names and focused instructions
var planner = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "ContentPlanner",
    Description = "Plans article structure and research needs",
    ChatOptions = new ChatOptions
    {
        Instructions = """
            Create a structured content plan with:
            1. Clear outline with main sections and sub-topics
            2. Specific research requirements for each section
            3. Target audience and appropriate tone
            4. Estimated word count and key messaging
            """
    }
});

// Avoid - generic names and vague instructions
var agent1 = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "Agent1",
    ChatOptions = new() { Instructions = "Do planning stuff" }
});

2. Implement Timeout Handling

Always use the hard timeout pattern for production workflows:

// Production-ready timeout handling
var workflowTask = harness.RunWorkflowTestAsync(adapter, testCase, options);
var hardTimeout = Task.Delay(options.Timeout ?? TimeSpan.FromMinutes(5));

if (await Task.WhenAny(workflowTask, hardTimeout) == hardTimeout)
{
    _logger.LogWarning("Workflow {WorkflowName} exceeded timeout {Timeout}", 
                      adapter.Name, options.Timeout);
    
    // Handle gracefully - don't fail tests in CI
    return new WorkflowTestResult
    {
        TestName = testCase.Name,
        Passed = false,
        ErrorMessage = "Workflow exceeded hard timeout",
        Duration = options.Timeout ?? TimeSpan.FromMinutes(5)
    };
}

var result = await workflowTask;

3. Layer AssertionsComprehensively

Start with basic structure, then add detailed validations:

// 1. Basic structure first
result.ExecutionResult!.Should()
    .HaveStepCount(expectedSteps.Length)
    .HaveExecutedInOrder(expectedSteps)
    .HaveNoErrors();
    
// 2. Add performance constraints  
result.ExecutionResult!.Should()
    .HaveCompletedWithin(TimeSpan.FromMinutes(3));
    
// 3. Add detailed per-executor validation
result.ExecutionResult!.Should()
    .ForExecutor("Planner")
        .HaveNonEmptyOutput()
        .HaveCompletedWithin(TimeSpan.FromSeconds(60))
        .And();
        
// 4. Add tool validation (if applicable)
if (result.ExecutionResult.ToolUsage != null)
{
    result.ExecutionResult.Should().HaveNoToolErrors();
}

// 5. Add cost/resource validation
result.ExecutionResult!.Performance!.Should()
    .HaveEstimatedCostUnder(maxExpectedCost);

4. Export Debug Information on Failures

[Fact]
public async Task ComplexWorkflow_Should_CompleteSuccessfully()
{
    var result = await harness.RunWorkflowTestAsync(adapter, testCase);
    
    // Export debug info on failure
    if (!result.Passed || result.ExecutionResult?.HasErrors == true)
    {
        var timestamp = DateTime.Now.ToString("yyyyMMdd-HHmmss");
        
        // Export Mermaid diagram
        var mermaid = WorkflowSerializer.ToMermaid(result.ExecutionResult!);
        var mermaidPath = $"failed-workflow-{timestamp}.mmd";
        File.WriteAllText(mermaidPath, mermaid);
        
        // Export timeline JSON
        var timeline = WorkflowSerializer.ToTimelineJson(result.ExecutionResult!);
        var timelinePath = $"failed-workflow-{timestamp}.json";
        File.WriteAllText(timelinePath, timeline);
        
        _testOutput.WriteLine($"Debug exports: {mermaidPath}, {timelinePath}");
    }
    
    result.Should().BeSuccessful();
}

5. Test Individual Steps in Isolation

Before testing complete workflows, validate individual agents:

[Fact] 
public async Task PlannerAgent_Should_CreateValidPlan()
{
    // Test individual agent before workflow integration
    var planner = CreatePlannerAgent();
    var input = "Write an article about AI agent testing";
    
    var response = await planner.InvokeAsync(input);
    
    Assert.NotNull(response);
    Assert.Contains("outline", response.Result, StringComparison.OrdinalIgnoreCase);
    Assert.Contains("research", response.Result, StringComparison.OrdinalIgnoreCase);
}

[Fact]
public async Task FullWorkflow_Should_IntegrateAgentsCorrectly() 
{
    // Test complete workflow after individual agents work
    var result = await harness.RunWorkflowTestAsync(adapter, testCase);
    result.Should().BeSuccessful();
}

Integration with CI/CD

JUnit XML Export

Export workflow evaluation results for CI/CD pipelines:

using AgentEval.Workflows.Serialization;

// Run multiple workflow tests
var testResults = new List<WorkflowTestResult>();
foreach (var testCase in testCases)
{
    var result = await harness.RunWorkflowTestAsync(adapter, testCase);
    testResults.Add(result);
}

// Export to JUnit XML for CI/CD
var report = new EvaluationReport
{
    Name = "Workflow Evaluation Tests",
    Results = testResults.Select(r => new TestResult
    {
        Name = r.TestName,
        Passed = r.Passed,
        Duration = r.Duration,
        ErrorMessage = r.ErrorMessage
    }).ToList()
};

using var outputStream = File.Create("workflow-test-results.xml");
await new JUnitXmlExporter().ExportAsync(report, outputStream);

GitHub Actions Integration

Example workflow for GitHub Actions:

name: Workflow Evaluation Tests

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  evaluate-workflows:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v4
    
    - name: Setup .NET
      uses: actions/setup-dotnet@v4
      with:
        dotnet-version: '8.0'
    
    - name: Run Workflow Evaluations
      env:
        AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
        AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
        AZURE_OPENAI_DEPLOYMENT: ${{ vars.AZURE_OPENAI_DEPLOYMENT }}
      run: |
        dotnet test tests/WorkflowEvaluationTests/ \
          --logger "junit;LogFilePath=workflow-results.xml" \
          --logger "console;verbosity=detailed"
    
    - name: Upload Test Results
      uses: dorny/test-reporter@v1.7.0
      if: always()
      with:
        name: Workflow Evaluation Results
        path: workflow-results.xml
        reporter: java-junit
        
    - name: Upload Workflow Diagrams
      uses: actions/upload-artifact@v4
      if: failure()
      with:
        name: workflow-debug-diagrams
        path: "*.mmd"

Performance Monitoring

Track workflow performance over time:

// Collect performance metrics
var metrics = new WorkflowMetrics
{
    WorkflowName = adapter.Name,
    ExecutionTime = result.Duration,
    StepCount = result.ExecutionResult?.Steps.Count ?? 0,
    TotalCost = result.ExecutionResult?.Performance?.EstimatedCost ?? 0,
    SuccessRate = result.Passed ? 1.0 : 0.0,
    Timestamp = DateTime.UtcNow
};

// Export to monitoring system
await MonitoringClient.RecordMetricsAsync(metrics);

// Set CI performance gates
if (metrics.ExecutionTime > TimeSpan.FromMinutes(5))
{
    throw new PerformanceException($"Workflow {adapter.Name} exceeded 5 minute SLA");
}

if (metrics.TotalCost > 0.50m)
{
    throw new CostException($"Workflow {adapter.Name} exceeded $0.50 cost limit");
}

Recording Workflows for CI/CD

Use WorkflowTraceRecorder to capture workflow executions for deterministic replay in CI — no LLM API calls needed:

// Record a real workflow execution
await using var recorder = new WorkflowTraceRecorder(adapter, "content_pipeline");
var result = await recorder.ExecuteWorkflowAsync("Write article about AI testing");
await recorder.SaveAsync("content-pipeline.trace.json");

// In CI — replay without API calls
var replayer = await WorkflowTraceReplayingAgent.FromFileAsync("content-pipeline.trace.json");
var replayResult = await replayer.ExecuteWorkflowAsync("Write article about AI testing");
// replayResult is identical to the original — zero cost, instant, deterministic

See Tracing for complete Record & Replay documentation.

See Also

Core Documentation:

• Architecture Overview - System architecture including workflow components
• Assertions - Complete assertion API with workflow examples
• Getting Started - Includes workflow evaluation quickstart

Related Evaluation Types:

• Conversations - Multi-turn conversation evaluation
• Tracing - Record and replay for deterministic workflow testing
• Stochastic Evaluation - Statistical workflow reliability
• Model Comparison - Compare models in workflow contexts

Advanced Topics:

• Red Team - Security evaluation for workflow endpoints
• Performance & Cost - Workflow performance optimization
• Export Formats - JUnit XML / SARIF / JSON export for CI/CD pipelines

Live Examples:

• Sample 09: Real MAF Workflow Evaluation - Sequential content pipeline (Planner → Researcher → Writer → Editor)
• Sample 10: Workflow with Tools - Tool-enabled trip planning workflow with function calling

Microsoft Agent Framework:

• MAF Official Documentation - Microsoft Agent Framework docs
• MAF Workflow Samples - Official MAF workflow examples