Table of Contents

Export Formats

AgentEval includes a complete export system for evaluation results, designed for CI/CD integration, reporting, and analysis.

Overview

All exporters implement the IResultExporter interface and can be created via the ResultExporterFactory:

using AgentEval.Exporters;

// Create by format enum
var exporter = ResultExporterFactory.Create(ExportFormat.Junit);

// Or by file extension
var exporter = ResultExporterFactory.CreateFromExtension(".json");

Available Formats

Format Extension Use Case ContentType
JSON .json Programmatic access, dashboards, APIs application/json
JUnit XML .xml CI/CD (GitHub Actions, Azure DevOps, Jenkins) application/xml
Markdown .md PR comments, documentation, GitHub rendering text/markdown
TRX .trx Visual Studio Test Explorer, Azure DevOps application/xml
CSV .csv Excel, Power BI, business intelligence tools text/csv
Directory (dir) Cross-run comparison, history, reproducibility (ADR-002) application/x-directory

Quick Start

// 1. Build an EvaluationReport
var report = new EvaluationReport
{
    Name = "Agent Quality Check",
    TotalTests = 10,
    PassedTests = 8,
    FailedTests = 2,
    OverallScore = 82.5,
    StartTime = DateTimeOffset.UtcNow.AddSeconds(-30),
    EndTime = DateTimeOffset.UtcNow,
    Agent = new AgentInfo { Name = "CustomerBot", Model = "gpt-4o" },
    TestResults = results // List<TestResultSummary>
};

// 2. Export to any format
var exporter = ResultExporterFactory.Create(ExportFormat.Junit);
await using var stream = File.Create("results.xml");
await exporter.ExportAsync(report, stream);

Format Details

JSON

Structured JSON with camelCase naming. Ideal for programmatic consumption.

var exporter = ResultExporterFactory.Create(ExportFormat.Json);
await using var stream = File.Create("results.json");
await exporter.ExportAsync(report, stream);

// Or export to string directly
var jsonExporter = new JsonExporter();
var json = await jsonExporter.ExportToStringAsync(report);

Output includes runId, stats, overallScore, agent info, and each test result with optional metricScores and metricsNotMeasured (a metric that ran but was not measured — an input it needs was not supplied, or, in a memory benchmark report, its category was skipped — with its reason; it has no score, so it is never in metricScores).

A value that was not recorded is written as null, never as 0: overallScore when no test was measured, durationMs when the run was not timed, stats.passRate for a report with no tests, and a test's score and durationMs when it produced no score or carries no timing. Each test also has notMeasured: true when it produced no verdict (the judge produced none, or the run errored), which is not a failure.

JUnit XML

Standard JUnit XML format compatible with all major CI/CD systems:

  • GitHub Actions: dorny/test-reporter@v1, EnricoMi/publish-unit-test-result-action
  • Azure DevOps: PublishTestResults@2 with testResultsFormat: 'JUnit'
  • Jenkins: Built-in JUnit plugin
  • GitLab CI: artifacts:reports:junit
  • CircleCI: store_test_results
var exporter = ResultExporterFactory.Create(ExportFormat.Junit);
await using var stream = File.Create("results.xml");
await exporter.ExportAsync(report, stream);

Tests are grouped by category into <testsuite> elements. Failed tests include <failure> elements with score and error details. A test that produced no verdict (TestResultSummary.NotMeasured: the judge produced none, or the run errored) is an <error type="NotMeasured"> with the reason, counted in errors: not a failure, and not a <skipped>, which CI reads as not run and keeps green. Metric scores are written to <system-out>, and a metric that was not measured as name: not measured — reason.

A time that was not recorded is left out rather than written as 0.000: on <testsuites> when the report was not timed, on a <testsuite> (the sum of its timed tests) when none of its tests was timed, and on a <testcase> with no timing. The overallScore property reads n/a when no test was measured.

Markdown

GitHub-flavored Markdown with tables, emoji status indicators, and configurable sections.

var mdExporter = new MarkdownExporter
{
    Options = new MarkdownExportOptions
    {
        FailuresFirst = true,          // Show failures at top
        IncludeFailureDetails = true,  // Detailed failure section
        IncludeMetricBreakdown = true, // Dynamic metric table
        IncludeFooter = true           // Run ID + timestamp
    }
};

// Export to string (ideal for PR comments)
var markdown = mdExporter.ExportToString(report);

// Or export to stream
await using var stream = File.Create("results.md");
await mdExporter.ExportAsync(report, stream);

The Markdown exporter renders:

  • Status header with ✅/❌ emoji: FAILED when a test failed, else ⚠️ INCOMPLETE when a test produced no verdict (with the count), else PASSED; the score reads n/a when no test was measured, and the duration not recorded when the report was not timed
  • Results table with score, status, and duration (a test with no verdict reads n/a and ⚠️ not measured; a test with no timing reads n/a for its time)
  • Optional failure details section
  • Optional metric breakdown table (dynamic columns from MetricScores and MetricsNotMeasured; a metric that was not measured reads not measured, one not requested for that test -)
  • Footer with run ID and timestamp

TRX

Visual Studio TRX format — native for .NET tooling and Azure DevOps.

var exporter = ResultExporterFactory.Create(ExportFormat.Trx);
await using var stream = File.Create("results.trx");
await exporter.ExportAsync(report, stream);

Uses deterministic GUIDs based on test names for reproducible output. Includes full TRX structure: TestRun, Times, ResultSummary, TestDefinitions, TestEntries, and Results. A test that produced no verdict has outcome Error (counted in error, not notExecuted), and the ResultSummary outcome is Failed when a test failed or produced no verdict. A UnitTestResult with no recorded timing has no duration and no endTime attribute, rather than 00:00:00.

CSV

Comma-separated values optimized for Excel and business intelligence tools.

var exporter = ResultExporterFactory.Create(ExportFormat.Csv);
await using var stream = File.Create("results.csv");
await exporter.ExportAsync(report, stream);

// Or export to string
var csvExporter = new CsvExporter();
var csv = await csvExporter.ExportToStringAsync(report);

Fixed columns: RunId, TestName, Category, Score, Passed, Skipped, DurationMs, Error, AgentName, AgentModel. A test that produced no score has not measured in Score, and a test with no timing an empty DurationMs: neither is written as 0.

Dynamic columns are appended for each unique key in MetricScores and MetricsNotMeasured (e.g., relevance, correctness). A cell holds the score, not measured when the metric ran without an input it needs, or is empty when the metric was not requested for that test — so a metric column can mix numbers and text (read it as text, or treat not measured as missing, not 0). Special characters (commas, quotes, newlines) are properly escaped per RFC 4180.

Directory (ADR-002)

Structured directory format for cross-run comparison, history tracking, and reproducibility. Produces multiple files per run instead of a single file.

ADR: 002-result-directory-structure.md

var exporter = new DirectoryExporter();
await exporter.ExportToDirectoryAsync(report, "./results/baseline");

// Or use the auto-generated directory name
var dirName = DirectoryExporter.GenerateDirectoryName(report);
// e.g., "2026-03-01_14-30-00_gpt-4o"
await exporter.ExportToDirectoryAsync(report, $"./results/{dirName}");

Library usage: DirectoryExporter retains the ADR-002 layout below. The CLI writes AEF as its run record; eval --output-dir selects an AEF destination. Benchmark JSON/Markdown/HTML/PDF reports remain separate exports, and agenteval render can re-render captured agentic reports from verified AEF at no LLM cost.

# Native AEF evidence from an evaluation run
agenteval eval --dataset cases.yaml --endpoint http://localhost:1234/v1 --model my-model --output-dir ./results/baseline

# A benchmark run is stored, not exported; re-render its report later (no LLM cost):
agenteval render --benchmark agentic --subject MyAgent

Each run produces a directory with:

File Format Purpose
results.jsonl JSON Lines One JSON line per test result (streaming-friendly, append-friendly); notMeasured is true for a test with no verdict; score and durationMs are null when not recorded; metricsNotMeasured names a metric that ran without an input it needs, with its reason
summary.json JSON Aggregate statistics with per-metric distribution (mean, min, max, stddev, percentiles) over the tests that measured it; metricsNotMeasured counts, per metric, the tests where it was not measured; overallScore, duration and stats.passRate are null when nothing was measured, the run was not timed, or there are no tests
run.json JSON Run metadata: agent info, environment, timestamp, duration (null when the run was not timed)
(original filename) (original format) Copy of original config/dataset file with filename preserved (when provided, for reproducibility)

The stream-based ExportAsync method writes summary.json content for IResultExporter compatibility. For full directory output, use ExportToDirectoryAsync.

The IResultExporter Interface

public interface IResultExporter
{
    ExportFormat Format { get; }
    string FormatName => Format.ToString();  // Default interface member
    string FileExtension { get; }
    string ContentType { get; }
    Task ExportAsync(EvaluationReport report, Stream output, CancellationToken ct = default);
}

The FormatName property is a default interface member that returns the enum name for built-in exporters. Custom exporters can override it to provide a meaningful string name for registry lookup.

What an exporter receives (and what it does not)

An exporter receives a flat EvaluationReport, the shape agenteval eval builds from its test harness and metrics: per test a 0–100 score, Passed, Skipped, an error, the output, metric scores and assertions. It does not receive the result model — EvalResult trees with MeasurementState, the warn / error / skipped / inapplicable labels, composite sub-results and judge provenance. Those come from the eval pipeline (IEval, CompositeEval, the bench commands), which does not call exporters.

Who calls a custom exporter: your code. agenteval eval --format accepts only the built-in names above; it does not look in IExporterRegistry. Register a custom exporter, resolve it from the registry and call ExportAsync yourself, as in the example below.

So a format that needs to keep "not measured" apart from "failed" cannot get it through IResultExporter. Take the EvalResult trees directly instead — from IEval.EvaluateAsync, or from a stored run — and serialise them against eval-result.schema.json (v1, embedded in AgentEval.DataLoaders). See ADR-034 for why the two are separate and the planned path to export the result model through a registry.

Third-party exporters

These are written and maintained outside AgentEval. Check each one's own documentation for what it supports.

Package What it does Version
EvalPort.AgentEval Writes AgentEval results in the EvalPort format and reads EvalPort suites as datasets. Besides IResultExporter and IDatasetLoader, it takes EvalResult trees directly (EvalPortDocuments.Build), so "not measured" stays apart from "failed" evalport-agenteval-dotnet/v0.1.1-beta, built against AgentEval 0.43.0-beta; not on NuGet, build it from the tag

Creating Custom Exporters

Implement IResultExporter to add new formats. Use the FormatName property for registry identification:

public class SarifExporter : IResultExporter
{
    public ExportFormat Format => ExportFormat.Json; // Closest built-in fallback
    public string FormatName => "sarif";             // Custom name for registry lookup
    public string FileExtension => ".sarif";
    public string ContentType => "application/sarif+json";

    public async Task ExportAsync(EvaluationReport report, Stream output, CancellationToken ct = default)
    {
        // Your serialization logic here
    }
}

Using IExporterRegistry (DI-Friendly)

The IExporterRegistry provides dynamic exporter lookup and registration, analogous to IMetricRegistry. Register custom exporters via DI:

// Register your custom exporter
services.AddSingleton<IResultExporter, SarifExporter>();
services.AddAgentEvalDataLoaders(); // Builds IExporterRegistry: built-ins + DI-registered exporters
                                    // (AddAgentEval() alone does not; AddAgentEvalAll() does)

// Resolve and use
var registry = serviceProvider.GetRequiredService<IExporterRegistry>();
var exporter = registry.GetRequired("sarif");
await exporter.ExportAsync(report, stream);

// List all available formats
foreach (var format in registry.GetRegisteredFormats())
{
    Console.WriteLine($"  {format}");
}

Built-in exporters (JSON, JUnit, Markdown, CSV, TRX, Directory) are pre-registered automatically. DI-registered exporters are added alongside them without overriding built-ins.

You can also register exporters manually at runtime:

var registry = serviceProvider.GetRequiredService<IExporterRegistry>();
registry.Register("powerbi", new PowerBIExporter());

Note: ResultExporterFactory.Create() and ResultExporterFactory.CreateFromExtension() continue to work for non-DI scenarios. The IExporterRegistry is the recommended DI-friendly alternative.

The EvaluationReport Model

var report = new EvaluationReport
{
    RunId = "auto-generated-8-char-hex",  // Auto-generated if not set
    Name = "Suite Name",
    StartTime = DateTimeOffset.UtcNow,
    EndTime = DateTimeOffset.UtcNow,
    TotalTests = 10,
    PassedTests = 8,
    FailedTests = 2,
    SkippedTests = 0,
    OverallScore = 85.0,
    Agent = new AgentInfo { Name = "Bot", Model = "gpt-4o" },
    Metadata = new() { ["environment"] = "staging" },
    TestResults = new List<TestResultSummary>
    {
        new()
        {
            Name = "tool_ordering_test",
            Category = "Agentic",
            Score = 95.0,
            Passed = true,
            DurationMs = 1200,
            MetricScores = new()
            {
                ["relevance"] = 92.5,
                ["correctness"] = 88.0
            }
        }
    }
};

Computed properties:

  • Duration — EndTime - StartTime, or null when EndTime is not after StartTime (the report was not timed)
  • PassRate — a percentage, or null for a report with no tests

OverallScore, TestResultSummary.Score and TestResultSummary.DurationMs are nullable too: null means not measured or not recorded, never 0. TestSummary.ToEvaluationReport() leaves OverallScore null when no test was measured, a test's Score null when it produced no verdict, and its DurationMs null when it carries no performance data. TestSummary.TotalDuration sums the timed tests (null when none was); TestSummary.TotalCost is complete or absent — null when any test was not priced, with UnpricedTests and TotalCostNotPricedReason saying which and why.

CI/CD Integration

GitHub Actions

- name: Run AgentEval
  run: dotnet test --logger trx --logger "junit;LogFilePath=results.xml"

- name: Publish Results
  uses: dorny/test-reporter@v1
  with:
    name: AgentEval Results
    path: results.xml
    reporter: java-junit

Azure DevOps

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/results.xml'

See Also

Edit this page