Table of Contents

CLI Reference

AgentEval ships a CLI for managing the .agenteval/ workspace from the terminal and CI/CD pipelines — and, via the gatekeeper verb group, a language-neutral runtime-policy service any process (Python, Node, bash, a CI step) can call for a versioned gate verdict + exit code.

Installation

# Recommended — install once, use anywhere
dotnet tool install --global AgentEval.Cli --prerelease

# Update later
dotnet tool update --global AgentEval.Cli --prerelease

# Or run from a cloned repo (contributor / development path)
dotnet run --project src/AgentEval.Cli -- <command>

After global install, the agenteval command is available system-wide. Requires .NET 8 SDK or later for the core surface; agenteval mc serve additionally requires .NET 10 because Mission Control depends on Hot Chocolate 16 + MapStaticAssets (net10-only). On .NET 8/9 installations, mc serve exits with a graceful "requires .NET 10" message rather than failing obscurely.

Examples below use the global agenteval form. To run from a cloned repo, substitute dotnet run --project src/AgentEval.Cli -- (note the trailing --).

Environment variables

The CLI honours the following process-level environment variables.

AI_INFERENCE_PROVIDER — which provider the CLI talks to

Every bench and calibrate command reaches its model through this selector. Set it to one of azure, bitdeer, openai, foundry or openai-compatible, then set that provider's variables:

AI_INFERENCE_PROVIDER Provider Variables it needs
azure Azure OpenAI AZURE_OPENAI_ENDPOINT + AZURE_OPENAI_API_KEY + AZURE_OPENAI_DEPLOYMENT
bitdeer Bitdeer AI Model Studio BITDEER_API_KEY (BITDEER_ENDPOINT, BITDEER_MODEL have defaults)
openai OpenAI OPENAI_API_KEY (OPENAI_BASE_URL, OPENAI_MODEL have defaults)
foundry Azure AI Foundry FOUNDRY_ENDPOINT + FOUNDRY_API_KEY + FOUNDRY_MODEL
openai-compatible any OpenAI-compatible host OPENAI_COMPATIBLE_ENDPOINT + OPENAI_COMPATIBLE_MODEL (OPENAI_COMPATIBLE_API_KEY optional; a keyless local host gets no-key-needed)

Leaving the selector unset is supported and backward compatible. The resolver then auto-detects, in this order: Azure OpenAI, Bitdeer, OpenAI, Foundry, OpenAI-compatible — the first one whose variables are all present wins. A machine that has only ever set AZURE_OPENAI_* therefore behaves exactly as it did before this variable existed.

Endpoint policy: an endpoint must be https, or http to loopback. Diagnostics name the variable and the reason, never its value, because a configured URL can carry a token in its user-info, query or fragment and the diagnostic reaches stderr.

Reasoning is off by default. Every chat client the CLI, the samples and the demos build from these variables sends a request that sets no reasoning level of its own with reasoning_effort: none. When the provider rejects that as an invalid request (HTTP 400 or 422) it sends it again at low, and when the provider rejects both (a model with no reasoning setting, such as gpt-4o) with no reasoning field, which is the provider's default. The level that worked is remembered per provider, endpoint and model for the rest of the process, so a rejected level costs one failed call per model. Code that sets ChatOptions.Reasoning keeps its own level. In your own code, chatClient.WithMinimalReasoning() (AgentEval.Core) does the same. The AEF dataset runner (agenteval run --plan) resolves the same none → low → provider-default sequence during readiness, with every attempt charged to its role's call and cost bounds. Cases then use the accepted setting for that role without renegotiating it. Each transport call makes one attempt; preparation reserves up to three accounted readiness attempts per role and suite. Why: on Bitdeer, GLM-5.3-Flash at its default reasoning spent two thirds of its output tokens thinking and took 2.2 times as long on an article-length answer (2026-10-21); Bitdeer rejects none and accepts low, which turns GLM's reasoning off. Results measured before 2026-10-21 ran at each provider's default.

Provider selection was added in v0.41.0-beta. Before that the CLI assumed Azure OpenAI.

AZURE_OPENAI_JUDGE_ENDPOINT, AZURE_OPENAI_JUDGE_API_KEY, AZURE_OPENAI_JUDGE_DEPLOYMENT

An optional judge override, independent of the selector above. When all three are set they win outright, so a capable grader can face a cheap subject in a single run. The same endpoint policy applies.

AGENTEVAL_ALLOW_STUB_JUDGE (retired)

Ignored since 0.43. Through 0.42 it let the bench commands, and calibrate, run with a placeholder judge that scored 75/100 with every criterion met, so the results and calibration figures measured no judge. There is no stand-in judge now: with no provider configured, every command that needs a judge exits 3. To try a command without a model, use --sut mock (see agenteval bench): it runs a built-in stand-in agent and a placeholder judge, says MOCK, exits 11 and writes nothing to .agenteval/.

Resolution order (exit codes per Exit codes):

  1. Test override (programmatic; not user-visible).
  2. All three AZURE_OPENAI_JUDGE_* set → that judge, whatever the selector says.
  3. A provider resolves with credentials → the real judge for that provider.
  4. Any provider variable or the selector is set, but a provider could not be built → exit 3 (RuntimeError), saying the provider is misconfigured.
  5. Nothing configured → exit 3, listing each provider and the variables it would need.

AgentEval__Root

Workspace-root override for processes that aren't launched from inside the workspace. Read by agenteval mc serve (the Mission Control host) and any program using AgentEvalServiceCollectionExtensions.AddAgentEvalAll(). Double-underscore is ASP.NET Core's hierarchical-key separator (AgentEval:Root in appsettings.json → AgentEval__Root as an env var).

ASPNETCORE_URLS

Honoured only when launching Mission Control directly (dotnet run --project src/AgentEval.MissionControl). agenteval mc serve forcibly binds to http://127.0.0.1:<port> and overrides this variable — there is no built-in auth in Phase 1, so the CLI hard-pins to loopback. To bind a broader interface (e.g. LAN), run the portal binary directly with your own ASPNETCORE_URLS and accept the trust trade-off.

Troubleshooting: --log-file <path>

A global option — available on every command, not just the ones shown below. Writes a human-readable, plain-text log of every LLM round-trip (request + response, including tool calls, usage, and finish reason) to <path>, separate from the command's normal stdout/stderr. Also captures the full exception (type + message + stack trace) for any request that fails, not just the short one-line summary the CLI prints to stderr by default.

agenteval eval --dataset my-data.jsonl --azure --deployment-name gpt-4o-mini --log-file trace.log
agenteval gatekeeper calibrate --gate judge:crescendo-trajectory-turn-shift --azure --deployment-name gpt-4o-mini --log-file calibrate-debug.log

Covers every LLM call the CLI makes for the invoked command — the agent/SUT under test, judge, attacker (RedTeam Crescendo/PAIR/TAP), and Copilot Studio's live connector all get logged when active.

⚠️ Contains raw, unredacted content. The log file includes the full text of every prompt and response — which can carry secrets, PII, or anything else present in your data or the model's output. --log-file is opt-in specifically for troubleshooting: turning it on means you want to see exactly what was sent and received. Never commit or share the resulting file. The file is overwritten on each invocation, so a fresh run always starts clean.

An unwritable --log-file path (missing parent directory, no permissions) never fails the command — it prints one warning to stderr and the invoked command runs exactly as it would without --log-file at all. Verbose logging is a debugging aid; it must never be why an otherwise-successful run fails.

A command line that does not parse exits 2 before --log-file or --capture-fixture is opened, so neither file is created or overwritten — see Exit codes.

Fixture capture

--capture-fixture <path> is a global option, available on every command. It writes one JSON line per LLM round-trip to <path>, for agenteval log-file to turn into a test fixture or to replay against another model. It is separate from --log-file, and the two can be passed together: --log-file is a log for reading, while each --capture-fixture line is a self-contained record of one round-trip, including every message the request sent.

agenteval eval --dataset my-data.jsonl --azure --deployment-name gpt-4o-mini --capture-fixture capture.jsonl

Each line has these fields:

Field Content
Index, Label The round-trip's position among those of the same client, and the client's role (agent, judge, …).
Kind response; error (the call threw); or abandoned (a streamed response the caller stopped reading before it ended).
TimestampUtc, ElapsedMs When the line was written, and how long the round-trip took.
Request The instructions, the options (temperature, max output tokens, model id, tool definitions) and every message sent, as role and text.
Response Text, tool calls, finish reason and token usage. Only on response lines.
Error The full exception text, stack trace included. Only on error lines.

Treat the file like a --log-file. Strings that match known credential formats are masked before each line is written; everything else — prompts, responses, personal data — is written as it was sent or received. Never commit or share the file. It is overwritten on each invocation, and an unwritable path prints one warning and the command runs without capture.

Commands

agenteval init

Initialize a starter evaluation dataset in the current directory.

Synopsis

agenteval init [--format yaml|json] [-o <path>] [--force]

What it does

Writes a sample dataset file for the legacy eval command surface. The default output is agenteval.yaml; pass --format json for a JSON starter, -o to choose a different file, and --force to overwrite an existing target.

Options

Option Description
--format <yaml|json> Output format. Default: yaml.
-o, --output <path> Output file path. Default: agenteval.{format}.
--force Overwrite an existing file.

Exit codes

Code Meaning
0 Dataset written successfully.
2 Invalid format or the target file already exists.

agenteval init-workspace

Initialize the canonical .agenteval/ workspace for the current solution.

Synopsis

agenteval init-workspace [--name <display-name>]

What it does

Walks up from the current directory until it finds a .sln, .slnx, or .git marker and treats that directory as the workspace root. Creates .agenteval/ if it does not exist, then writes:

  • solution.json — solution-level identity: a random UUID, the display name, and schemaVersion: "1.0".
  • README.md — overview of the workspace layout.
  • .gitignore — excludes per-run artifacts and red-team outputs from source control.

If solution.json already exists, the command reports that the workspace is already initialized and exits cleanly.

Options

Option Description
--name <display-name> Display name to record in solution.json. Defaults to the directory name of the solution root.

Exit codes

Code Meaning
0 Initialized successfully (or already initialized).
1 Could not locate a solution root.

agenteval eval

Evaluate an AI agent against a dataset.

Synopsis

agenteval eval --dataset <path> --endpoint <url> [--model <name>] [--azure --deployment-name <name>] [options]

What it does

Loads a YAML, JSON, JSONL, CSV, or TSV dataset, evaluates the agent, and exports results as JSON, JUnit/XML, Markdown, TRX or CSV, with AEF as the run record. It supports stochastic reruns (--runs), LLM-as-judge, named metrics (--metrics), and AEF evidence under --output-dir.

Key options

Option Description
--dataset <path> Required. Input dataset file: JSON, JSONL, CSV/TSV or YAML. Field names match in any spelling (expected_output, expectedOutput, ExpectedOutput).
--endpoint <url> / --azure / --deployment-name <name> Choose OpenAI-compatible or Azure OpenAI mode.
--model <name> Required for non-Azure endpoints.
--api-key <key> API key or environment variable fallback.
--sut copilot-studio Evaluate a live Microsoft Copilot Studio agent instead of --endpoint/--azure — bring your own dataset (prompts + judge criteria); requires --copilotstudio-config/--i-understand-live-side-effects. See Copilot Studio.
--system-prompt / --system-prompt-file Set the agent system prompt inline or from file. A --system-prompt-file that does not exist is ignored.
--temperature <value> Sampling temperature sent with every agent call. Omitted: nothing is sent and the provider's default applies. Given: the value is sent as given, 0 included. 0 narrows sampling; it does not guarantee identical outputs.
--max-tokens <n> Maximum output tokens per agent call. Omitted: nothing is sent.
--metrics <list> Comma-separated named metrics to score in addition to the normal pass/fail check (e.g. llm_relevance,code_tool_success). Each is scored against the same captured response; the agent is not called again. Omitted: no named metric is scored. An unknown name fails before any network call. Accepted names: llm_relevance, llm_faithfulness, llm_context_precision, llm_context_recall, llm_answer_correctness, llm_groundedness, llm_coherence, llm_fluency, llm_bias, llm_misinformation, llm_toxicity, llm_task_completion, code_tool_success, code_tool_efficiency, code_toxicity, code_skill_disclosure_efficiency, embed_answer_similarity, embed_response_context, embed_query_context. agenteval list --type metrics prints these and also names --metrics does not accept, each marked [library only: not available via --metrics]: code_tool_selection, code_tool_arguments, code_recall_at_k, code_mrr and ConversationCompleteness. They need input --metrics has no source for (expected tools, a K; ConversationCompleteness scores a multi-turn conversation, not a single response), so construct them in code. LLM-based (llm_*) names need --judge, or on the --endpoint/--azure path fall back to the agent's own model; code-based (code_*) names need neither. Embedding names (embed_*) use the environment-selected provider and its embedding model, independently of explicit subject/judge flags; see embedding configuration and usage evidence. A requested metric that is not measured on some case (its judge call failed, its reply could not be read, the case lacked an input it needs) makes a run in which nothing failed exit 11, never 0. Ignored, with a warning, when --runs is greater than 1.
--runs <N> Runs per test case. Default 1. Must be at least 1; greater than 1 is stochastic mode (below), which needs at least 3 runs.
--success-threshold <0..1> Stochastic mode only: the share of a test case's runs that must pass for the test case to pass. Default 0.8. Not used, and not checked, when --runs is 1.
--judge / --judge-model Separate LLM-as-judge endpoint/model.
--format <fmt> Export format. Default json. In stochastic mode, one entry per test case (below).
-o, --output <path> Output file for single-file formats. Default: stdout.
--output-dir <path> Sealed AEF evidence destination. Cannot conflict with --no-aef or a different --aef directory.
--save-golden <path> Save this run as a golden trace: each test case's verdict, output, and tool calls with their arguments, as JSON to commit beside the dataset (below).
--golden <path> Compare this run with a golden trace and exit on regressions only (below).
--fail-on-tool-change With --golden, also exit 1 when a test case's tool calls changed.
--quiet Suppress the header, progress, summary, and the --metrics and --sut warnings. Errors are still printed.
--aef <dir> / --no-aef Where the sealed AEF run goes: by default a workspace's .agenteval/aef; --aef writes it anywhere and fails the command (exit 3) if it cannot; --no-aef writes none. See Runs from eval, redteam and bench.
--subject <kind:name> / --subject-version <v> The subject the AEF run names, such as agent:support-bot (default model:<model>), and its exact version.

Stochastic mode (--runs greater than 1)

Each test case is run N times. A test case passes when the share of its runs that passed is at least --success-threshold, and the command exits 1 when any test case fails. A run that errored or got no verdict from the judge is not measured: when such runs could still decide whether the threshold is met (undecided), or no run was measured, the test case has no conclusive verdict, and the command exits 11 if no test case failed. The stochastic runner needs at least 3 runs, so --runs 2 is a usage error (exit 2), as are 0 and negative values; in this mode a --success-threshold outside 0–1 is a usage error too. These checks run before anything is loaded or called.

A table and a pass/fail line per test case, and a closing summary, are printed to stderr. The export (--format, -o) has one entry per test case, not one per run: it passes or fails on the pass rate, and its score is the mean over the runs. Every format carries the runs as four metric columns: stochastic_runs, stochastic_runs_passed, stochastic_pass_rate (0–100) and stochastic_score_sd. A failed test case's message names its pass rate and the threshold (3 of 5 runs passed (60.0%), below the 80% threshold.). JUnit system-out and TRX stdout also list each run with its score or error and the confidence interval for the mean. The report's name ends in (stochastic, N runs per test), and Mode, RunsPerTest and SuccessThreshold are recorded in the JSON export's metadata. AEF (--aef or --output-dir) retains trials and rollups separately. Through 0.43 this mode wrote no export at all. --metrics is ignored in this mode, with a warning.

Golden traces (--save-golden, --golden)

A golden trace is a saved run: for each test case, whether it passed, its score, the agent's output, and the tool calls it made in order, with their arguments (object keys sorted, so key order is not a change). Save one from a run you trust and commit it:

agenteval eval --dataset cases.yaml --endpoint <url> --model <name> --save-golden golden/cases.trace.json

Later runs compare against it:

agenteval eval --dataset cases.yaml --endpoint <url> --model <name> --golden golden/cases.trace.json

Each test case is reported on stderr as regressed (passed in the golden trace, fails now), improved, tools changed (a tool added, dropped or reordered, or called with other arguments), output changed, unchanged, added, removed or NOT MEASURED (the golden run or this one produced no verdict for it: the judge produced none, or the run errored; its placeholder fail is neither a regression nor an improvement). Test cases are matched by name. Output is compared after trimming and normalising line endings. Model output varies from run to run, so an output change is reported and does not fail the run. When either run recorded no tool data, tool calls are not compared.

With --golden the exit code follows the comparison, not the raw verdicts: 1 when a test case regressed (or, with --fail-on-tool-change, called different tools), 11 when none did but a test case is not measured, 0 otherwise. A test that already failed in the golden trace does not fail the build. Regressions and test cases not measured are printed even with --quiet. Give both options to compare and then update the file. A golden trace records one run per test case, so neither option combines with --runs above 1. A --golden file that is missing or unreadable is a usage error, raised before any agent call. In code, the same is GoldenTrace.FromResults(results) and GoldenTraceComparer.Compare(golden, current) in AgentEval.Snapshots.

With --sut

The target configures its own model, so --temperature, --max-tokens, --system-prompt and --system-prompt-file are not applied. A warning names the ones given (not printed with --quiet).

Exit codes

Code Meaning
0 Every test case passed. With --golden: no test case regressed (and, with --fail-on-tool-change, none changed its tool calls).
1 At least one test case failed — in stochastic mode, its pass rate was below --success-threshold even counting every run that was not measured as a pass. With --golden: a test case regressed, or changed its tool calls under --fail-on-tool-change.
2 Usage error: a missing --dataset, an unknown option, a value that does not parse (such as --runs abc), --runs below 1, a --runs or --success-threshold value stochastic mode does not accept, a --golden file that is missing or not a golden trace, --fail-on-tool-change without --golden, or a golden-trace option with --runs above 1.
3 Runtime or configuration error — including a missing --endpoint/--azure/--model, a dataset that does not exist or is empty, an unknown --metrics name, and an unknown --format, which is reported only after the evaluation has run.
11 No conclusive verdict, and no test case failed. In stochastic mode: a test case's verdict is undecided (runs that errored or got no judge verdict could still decide it) or not measured (no run was). With --metrics: a requested metric was not measured on some case (a judge call that failed, a reply that could not be read, an input the case lacked), so such a run never exits 0. With --golden: no test case regressed, but one has no verdict in the golden trace or in this run (NOT MEASURED), so the comparison cannot say whether it regressed.

agenteval migrate

Migrate legacy AgentEval output paths to the canonical .agenteval/ layout. Dry-run by default; pass --apply to commit changes.

Synopsis

agenteval migrate [--apply] [--root <path>]

What it does

Walks the workspace looking for three legacy patterns and reports (or moves, with --apply) each to its canonical location:

  1. Uppercase .AgentEval/ (Windows-collapsed casing) → lowercase .agenteval/ (preserves audit-chain integrity by moving in-place on the same volume).
  2. TestResults/traces/*.json legacy trace dumps → subjects/<kind>/<name>/runs/<runId>/traces/agent-trace.json per discovered subject (file is renamed to the canonical name).
  3. Flat .agenteval/benchmarks/ outside the per-subject hierarchy → subjects/<kind>/<name>/benchmarks/....

The dry-run output lists each move as MOVE <src> → <dest> so you can preview before committing. --apply performs the moves; --root <path> lets you target a specific workspace explicitly instead of the auto-detected one.

Options

Option Description
--apply Commit the moves. Without it, the command only prints what it would do.
--root <path> Workspace root path. Default: auto-detected.

Exit codes

Code Meaning
0 Migration plan printed (dry-run) or applied (--apply).
1 Could not locate a workspace root, or an I/O error occurred during a move.

agenteval doctor

Validate the .agenteval/ workspace structure and content hashes.

Synopsis

agenteval doctor

What it does

Performs five checks in sequence:

  1. solution.json — Verifies that schemaVersion, id (non-empty GUID), and name are all present and well-formed.
  2. Subject-name consistency — For each subject folder under subjects/agents/ and subjects/workflows/, verifies that the sanitized name field inside subject.json matches the folder name on disk.
  3. Per-run content hashes — For each run with a manifest.json, recomputes the SHA-256 hash over the run's summary, sorted scenario results, and optional trace, and compares it against the stored contentHash.
  4. Compliance evidence audit chain — For each evidence.json under compliance/, verifies that sourceRun.manifestHash matches the contentHash recorded in the source run's manifest.json.
  5. Stray output paths — Detects accidentally-created folders that shadow the canonical layout (.AgentEval/ with mixed case on case-sensitive filesystems, stray TestResults/traces/, or a flat .agenteval/benchmarks/ outside the per-subject hierarchy) and reports them as errors so they can be removed or merged.

After all checks, prints a summary line:

Errors: N | Warnings: N | OK: N

Example output (clean workspace)

✔ solution.json OK
✔ Run 3f8a1b2c (subject: TravelAgent)
✔ compliance/GDPR/TravelAgent/2026-04-10_14-32-00/evidence.json

Errors: 0 | Warnings: 0 | OK: 3

Example output (issues found)

✔ solution.json OK
✖ Hash mismatch in run 3f8a1b2c (subject: TravelAgent).

Errors: 1 | Warnings: 0 | OK: 1

Exit codes

Code Meaning
0 No errors found.
1 Could not locate a solution root or .agenteval/ is missing.
2 One or more validation errors found.

Warnings (e.g. a subject folder with a missing subject.json) do not affect the exit code.

agenteval compare

Scenario scores and deltas are nullable. If either side did not measure a scenario, the table shows n/m for that score and n/a for its delta. JSON includes explicit null baseline, candidate and delta fields where absent. The library's RunComparison.MeanScoreDelta is null when no matched pair was measured; a genuinely zero delta stays zero. Comparability checks and regression exit rules still apply.

Compare two AEF or legacy store runs, and print score deltas only when their recorded comparison facts agree.

Synopsis

agenteval compare --baseline <run-dir> --candidate <run-dir> [--strict] [--json] [--fail-on-regression]

What it does

Accepts native AEF run directories, legacy store run directories, copied legacy run folders and directories containing legacy scenario JSON files. A legacy run is converted through the store-v1 converter into temporary AEF evidence; its original files are unchanged. Pointing at its scenarios/ folder also works. Both inputs are verified before comparison, and invalid AEF evidence is refused. The report includes integrity, run status and whether evidence was imported. An import seal protects the converted evidence from import onward; it does not establish that the original run was sealed when it executed. Signature trust is not checked here. The report lists importer-supplied header fields. Without a legacy manifest, original execution identity, timing and completion are unknown: the temporary import uses a new ID, an unknown subject and explicit placeholder timestamps, with an aborted status explaining that missing evidence. A malformed manifest is refused. Copied folders use a bounded snapshot of ordinary files; links and special files are refused.

Comparison uses final observations, excluding individual trial rows. Native composite evaluations select their evaluation leaves; performance benchmarks select the whole benchmark score and retain component readings as supporting evidence. A readable AEF run still needs recorded, unambiguous comparison facts to produce deltas. The historical eval --output-dir report layout (results.jsonl, summary.json, run.json) is not a legacy scenario-store folder.

The comparison is refused — the reasons are printed and no delta is — when:

  • the two runs do not hold the same set of scenario ids, or share none;
  • either run recorded no comparability facts for a shared scenario (runs written before those facts existed); or
  • for a shared scenario, one of these axes differs between the runs, or only one run recorded it: evalKey, evalVersion, effectiveBar (the pass bar applied, compared exactly), evaluationPolicyDigest (scoring configuration and measurement protocol), judge (model, deterministic or unknown instrument), judge.modelId, judge.rubricDigest, and stimulus (a hash of what the scenario was asked);
  • the selected scores have different recorded metric identities, kinds, directions, scales or units.

An axis that neither run recorded is not counted as a match: it is printed as a blind spot, before the deltas. --strict refuses on it instead. When neither run used a judge, judge matches but judge.modelId is such a blind spot, so --strict refuses two runs that were both graded without a judge.

Performance policy evidence includes normalization budgets, sampling settings, aggregation and the frozen pricing schedule. It excludes subject identity and observed outcomes: different agents can be compared under the same policy. Older runs without a policy digest expose that absence; --strict refuses it. A digest recorded on only one side blocks even without strict mode. Different-policy trials cannot be rolled into one comparison instrument. Stimulus digests in AEF require content capture to be enabled; missing or heterogeneous stimuli remain unknown.

When the runs are comparable, the report lists each shared scenario's baseline score, candidate score and delta, then the mean delta and how many scenarios recovered (failed, then passed) or regressed (passed, then failed). A delta too small to show at four decimal places is printed in scientific notation rather than as 0.0000. Two findings are printed without blocking the comparison: scenarios where a run recorded no usable chance floor, so the delta cannot be read against chance, and scenarios graded by a judge running the subject's own model.

Options

Option Description
--baseline <path> Required. The baseline run directory.
--candidate <path> Required. The candidate run directory.
--strict Also refuse when an axis was recorded by neither run.
--json Print the comparison as JSON on stdout instead of the report. deltas is left out when the comparison is refused; when it is not, recovered, regressed and regressedScenarios (the ids) are included.
--fail-on-regression Exit 1 when the runs are comparable and a scenario the baseline passed fails in the candidate, and name those scenarios. For a CI step that must fail on a regression; without it, a comparable result exits 0 whatever it shows.

Exit codes

Code Meaning
0 Comparable; the deltas are printed.
1 With --fail-on-regression: comparable, and at least one scenario regressed (passed in the baseline, did not pass in the candidate: failed, errored or not run). A scenario not measured in the baseline cannot regress, so it does not set this exit code even when the candidate fails it.
2 A path is missing or not a directory, holds no scenario files, or holds a file that is not a readable scenario; or a run repeats a scenario id.
13 Incomparable; the reasons are printed and no delta is.

agenteval bench

Run benchmark families against a subject (agent or workflow). The benchmark registry now includes GDPR, EU AI Act, Agentic, OWASP, MITRE, NIST, Performance, LongMemEval, TypedMemEval, Memory, Trace Fidelity, and AutoAudit. Results flow into .agenteval/ so Mission Control and agenteval doctor can read them (except autoaudit, which prints its report and writes it with --out).

Synopsis

agenteval bench --list
agenteval bench <family> [family-specific options]
agenteval bench gdpr calibrate [--root <path>] [--out <path>] [--decisions]
agenteval bench eu-ai-act calibrate [--root <path>] [--out <path>] [--decisions]
agenteval bench agentic calibrate [--root <path>] [--out <path>] [--records <path>] [--limit <n>]

Families

GDPR and EU AI Act accept --runs N (at least 1, default 1). Repeated runs use majority voting, retain each trial in AEF, and persist the voted result for reporting. Ties choose the more severe label; a pass supported by only some requested trials becomes a warning. --sut mock still runs only its demonstration.

Family Purpose
gdpr GDPR compliance benchmark.
eu-ai-act EU AI Act compliance benchmark.
agentic Agentic tool-use benchmark family.
owasp OWASP LLM Top 10 red-team benchmark.
mitre MITRE ATLAS red-team benchmark.
nist NIST AI RMF-style red-team benchmark.
perf Latency / throughput / cost benchmark.
longmemeval Long-context memory benchmark.
typedmemeval TypedMemEval v5 (AgentEval) — one vertical per run: prospective, episodic, arithmetic, working-memory or forgetting.
memory Memory retention / cross-session benchmark.
trace-fidelity Chat-boundary vs agent-boundary trace reconciliation.
workflow-trace-fidelity Per-executor workflow ledger (tokens + finish reason) vs chat-boundary truth.
autoaudit Glass Box auto-audit of the configured models (or --models a,b): one support task each, ranked on honesty, safety and cost. Without a provider it refuses; --sut mock runs the scripted showcase, labelled MOCK.

Notes

  • agenteval bench --list prints the registry-backed family catalog.
  • AEF runs: gdpr, eu-ai-act, agentic, owasp, mitre, nist, perf, memory, longmemeval, typedmemeval, autoaudit, trace-fidelity and workflow-trace-fidelity write each run as a sealed AEF run into the workspace's .agenteval/aef (or --aef <dir>; --no-aef writes none; --subject-version records the subject's exact version). Autoaudit applies the supplied version to every audited model:<name>; use separate invocations when versions differ. A real target is recorded as live, a graded --response or trace as replayed. See Runs from eval, redteam and bench. Saved runs without an exact subject version cannot satisfy a checkpoint lane; native commands print a note after writing them. Eval and red-team suppress this informational note with --quiet.
  • Exit codes: bench <family> and bench <regulation> calibrate return 9 (FAIL), 10 (WARN — bench <family> only), or 11 (indeterminate) for a benchmark gate outcome, and 3 if the judge fails to configure — see Exit codes. A missing required option (--subject, and for some families --input, --vertical, --agent-trace/--chat-trace or --workflow-trace), an --evidence-detail value other than references or content, and agenteval bench with no family and no --list return 2.
  • Compliance and agentic families support calibration helpers where available. Each calibration report names the judge's provider and model in its header.
  • bench gdpr calibrate --decisions and bench eu-ai-act calibrate --decisions grade the golden datasets with the decision model (TypeSafe Jev) instead of the generative judge. The transport is read from TYPESAFE_API_KEY or OPENROUTER_API_KEY (JEV_TRANSPORT forces one, JEV_MODEL pins a build); with neither configured the command exits 3. The report header names the provider (TypeSafe or OpenRouter) and the model that was requested, marked as requested, because the provider may serve a different build under that name. It never contains the key or the endpoint.
  • typedmemeval takes --vertical <prospective|episodic|arithmetic|workingmemory|forgetting> and --subject, both required — the verticals measure different mechanisms, so there is no default. Corpora are embedded (no download, no dataset path); a real model is required, from whichever provider AI_INFERENCE_PROVIDER selects (the AZURE_OPENAI_* trio when it is unset), and there is no stub fallback. It prints the typed outcome vector with every denominator and gates on nothing: the family publishes no pass threshold, so a run exits 0 when it measured anything and 11 (GateIndeterminate) when it measured nothing, and its run summary is recorded as WARN (indeterminate) rather than PASS/FAIL. Cite results as TypedMemEval-<Vertical> v5 (AgentEval) — never summed or averaged with LongMemEval numbers. --vertical prospective additionally requires the agent under test to implement ITimestampedHistoryInjectableAgent; the run refuses before its first provider call otherwise.
  • Family-specific options and presets are documented under Benchmarks and the family pages in the TOC.
  • For the Trace Fidelity and AutoAudit families, see the historical design docs under docs/glassbox-history/ (linked in the TOC under Resources).
  • Every bench family that grades an agent needs a real target, or it refuses (exit 2). owasp/mitre/nist/perf take --from-env (the configured provider; --azure-from-env, its old name, still works), --sut copilot-studio (same flags as eval/redteam) or a generic --endpoint <url> --model <name> [--api-key <key>] OpenAI-compatible endpoint. gdpr/eu-ai-act take --from-env or --sut copilot-studio (each scenario's prompt is sent to the live agent), or the agent's real answer with --response/--response-file plus the --input it answered. agentic grades a supplied --response/--response-file with its --input, plus --reference/--reference-file (the expected answer) and --context/--context-file (the retrieved context) for the checks that grade against them (rag-quality); without them those checks report not measured. --sut mock runs a built-in stand-in instead of an agent: the run says MOCK, exits 11 whatever it scores, and nothing is written to .agenteval/. Through 0.42 these commands quietly fell back to a stand-in and stored the result as a measurement.
  • owasp/mitre/nist grade judge first, as redteam --judge does. The judge model comes from the environment (the AZURE_OPENAI_JUDGE_* override if set, otherwise the provider AI_INFERENCE_PROVIDER selects); there is no option to pick it. With no provider configured the command exits 3; --sut mock needs none and grades with the oracles alone. Before the scan the command makes one short call to the judge and exits 3 if it does not answer. The semantic attacks are graded by Composite Judges (several judge calls per probe); the other attacks by their per-attack oracle, which asks the judge only when it is inconclusive, and the judge may then only raise the probe to "attack succeeded". If a judge call fails during the scan, or the scan runs out of time, the run is INCOMPLETE: stored as WARN and exit 11, never a pass — unless what it did measure already fails it, which stays a fail (FAIL, exit 9). See the OWASP and MITRE pages for probe counts.

agenteval list

List the legacy command-surface catalogues used by eval / redteam.

Synopsis

agenteval list [--type metrics|attacks|exporters|datasets]

What it does

Prints the available metrics, attack types, export formats, and dataset formats. With no filter it prints all four catalogues.

The metrics catalogue marks every name that eval --metrics cannot resolve with [library only: not available via --metrics]. Whether a name carries the marker is read from the same table --metrics resolves against, so the listing cannot offer a name the command then refuses. Marked names need input --metrics has no source for (expected tools, a K); construct them in code.

Options

Option Description
--type <metrics|attacks|exporters|datasets> Print a single catalogue instead of all four.

Exit codes: 0 when the catalogue is printed; 2 for an unknown --type.

agenteval redteam

Run low-level red-team scans against an agent. This is the fully parameterised scanner surface; the bench owasp and bench mitre families wrap curated presets around it.

Synopsis

agenteval redteam [--azure] [--endpoint <url>] [--model <name>] [--deployment-name <name>] [--attacks <list>] [--format <fmt>] ...

Key options

Option Description
--azure / --endpoint / --deployment-name Azure OpenAI mode.
--endpoint / --model OpenAI-compatible mode (OpenAI, Ollama, Groq, vLLM, LM Studio, etc.).
--sut Built-in target instead of an endpoint: gatekeeper-demo (the Gatekeeper demo: the configured model behind the gate, or a labelled scripted model when no provider is configured) or copilot-studio (a live Microsoft Copilot Studio agent).
--scripted With --sut gatekeeper-demo: run the scripted model even when a provider is configured. Deterministic and free; use it for stable CI baselines. A baseline taken on one model is refused against a run on another (exit 3). See Attack the gate. With --attacks memory-poisoning: run a scripted worst-case model instead of --endpoint/--model; it saves what it is told and acts on recalled poison (it shows the gates, not a model).
--attacks Comma-separated attack list; --pack imports external benchmark packs. memory-poisoning runs on its own: the memory-security corpus against the model you name, behind AgentEval's memory protection (see Memory poisoning); it takes --memory-trials <n>, --format markdown\|json, --fail-on vuln\|never and --timeout-per-probe (each model call), refuses the probe-scan options (--sut, --intensity, --system-prompt, --pack, …) rather than ignoring them, and exits 0 pass, 1 a security check failed, 11 one could not be measured and none failed.
--transform Also run every single-turn probe encoded: codecs (base64, rot13, hex, …) or a group (reversible, lossy, all). The plaintext probes still run; each codec multiplies the probe count. See Transform pipeline.
--judge / --attacker Separate judge/attacker models for LLM-as-judge and attacker-LLM flows.
--format / -o Export format and output destination.
--baseline, --save-baseline, --fail-on Regression gating for CI.
--calibration Relative scoring against a reference cohort.
--explain Attach an LLM rationale to each finding (requires --judge).
--benign-controls Also run benign look-alike requests and report over-refusal beside the attack success rate, graded by the over-refusal judge (needs --judge; does not change the verdict).
--aef <dir> / --no-aef Where the sealed AEF run goes: by default a workspace's .agenteval/aef; --aef writes it anywhere and fails the command (exit 3) if it cannot; --no-aef writes none. See Runs from eval, redteam and bench.
--subject <kind:name> / --subject-version <v> The subject the AEF run names, such as agent:support-bot (default model:<model>), and its exact version.

For the full flag matrix and examples, see Red Team Security. Exit codes: see Exit codes.

agenteval gatekeeper

Run Gatekeeper runtime-enforcement gates from the terminal or any language — the same policy you red-team with, exposed as a language-neutral policy service. A process pipes a JSON payload to gatekeeper inspect and gets a versioned verdict JSON + an exit code; the deterministic gates need no credentials and are byte-stable, so they drop straight into CI.

Synopsis

agenteval gatekeeper list-gates [--json] [--phase inspect|serve|all]
agenteval gatekeeper inspect   --gate <id> [--input <file.jsonl>] [--policy block|warn] [gate flags] [model flags]
agenteval gatekeeper calibrate --gate judge:<axis> <model flags> [--certify]

Subcommands

Subcommand Purpose
list-gates List every gate, its state class, whether it needs a model, and its span policy.
inspect Evaluate one JSON payload (stdin) or one per line (--input JSONL) against a gate/panel; emits a versioned verdict + exit code.
calibrate Score a judge:<axis> against its gold set + keyword-oracle baseline; --certify writes the certificate the honesty guard reads.
serve Reserved for stateful accumulator gates — not implemented.

judge:* gates read/write a per-model calibration certificate under .agenteval/gatekeeper/certs/ (override with --cert-dir); the deterministic and tool gates are credential-free and CI-safe. For the full flag matrix, the verdict JSON contract, and the honesty guard, see Gatekeeper from any language.

agenteval skills scan

Static, offline compliance scan of a directory of MAF Agent Skills — no model call, no credentials. Reaches the same SkillComplianceValidator/MafSkillScanner library code the Agent Skills evaluation suite ships, from the CLI.

Synopsis

agenteval skills scan <path> [--format console|markdown|json] [-o|--output <file>] [--fail-on-noncompliant]
                              [--write-baseline] [--baseline-root <dir>] [--repo] [--check-baseline]
                              [--save-manifest-baseline <file>] [--manifest-baseline <file>] [--baseline-note <text>]

What it does

Walks <path> for SKILL.md-rooted skill folders (the same convention MAF's own AgentFileSkillsSource discovers by), checks each against the GA SKILL.md authoring rules (name/description/compatibility) plus governance flags (script-execution review, untrusted resource sources, experimental allowed-tools), and renders a report. v1 is compliance-only — the composite Skill Health & Security Index is library-only for now (see Agent Skills). Three additional governance signals ride along in the same report: cross-location content drift (--repo/scan-workspace only, always on), trust-on-first-use reputation matching (--check-baseline, opt-in), and manifest hash-pin drift against an explicit trust-time pin (--manifest-baseline, opt-in) — see Agent Skills for how each works.

When a skills-lock.json (the project lock file skill installers such as ChilliCream's skills CLI write) is in <path> or a parent of it up to the repository root, each finding names where its skill came from (→ from chillicream/agent-skills@a1b2c3d, in every format), and --write-baseline stores the source and ref on the snapshot. Nothing is fetched: it is a pointer for a person to check upstream, and it never changes a finding or the exit code.

Options

Option Description
<path> Required, positional. Directory containing one or more skill folders (a REPO ROOT when --repo is set).
--format <fmt> console (default), markdown, or json.
-o, --output <path> Write the rendered report to a file instead of stdout.
--fail-on-noncompliant Exit 1 when the scan finds a High-severity finding. Default off (informational-only). Cross-location drift (Medium) and previously-vetted matches (Low) never trigger this; a manifest-baseline drift finding (High) DOES.
--write-baseline Capture a timestamped baseline snapshot (structural fingerprint + full file-content hash per skill) into the baseline ledger. See agenteval skills baseline below.
--baseline-root <dir> Baseline ledger root directory. Default .agenteval/skills-baselines. Used by both --write-baseline and --check-baseline.
--repo Treat <path> as a repo root: scan every known skill-directory convention found under it (.claude/skills, .agents/skills, .windsurf/skills, ...) and aggregate the results into ONE combined report, instead of treating <path> itself as one skill directory. Per-convention breakdown (found/not-present, skill/finding counts) prints to stderr. When two or more conventions define the same skill name with DIFFERENT content, a CrossLocationContentDrift finding is added automatically.
--check-baseline Trust-on-first-use: compare each scanned skill's content hash against the baseline ledger's history. A match against any prior snapshot for the same name adds an informational MatchesPreviouslyVettedCopy finding. Meaningless on a first-ever scan (no history yet) — pair with --write-baseline on earlier runs.
--save-manifest-baseline <file> Capture a trust-time hash-pin of every scanned skill's manifest content (name, description, resource/script inventory, allowed-tools, compatibility) to this JSON file. A SINGLE pinned file, distinct from --write-baseline's multi-snapshot ledger — mirrors the RedTeam baseline/diff CI pattern: commit this file, then re-check future scans against it.
--manifest-baseline <file> Check every scanned skill against a --save-manifest-baseline file. A skill whose manifest content changed since the pin was captured is reported as a High-severity ManifestChangedSinceBaseline finding — a possible rug-pull.
--baseline-note <text> Optional human note saved alongside --save-manifest-baseline (e.g. who approved it, why).

Exit codes

Code Meaning
0 Scan completed (compliant, or --fail-on-noncompliant not set).
1 --fail-on-noncompliant was set and a High-severity finding was found.
3 Runtime error (e.g. <path> does not exist).

agenteval skills scan-workspace

Agent Skills Wave 3a — filesystem-only, credential-free scan across a folder of already-cloned repos. Every immediate (non-hidden) subdirectory of <path> is treated as one repo and scanned with the same pipeline scan --repo uses, then combined into one report. No GitHub/GitLab API, no token — the operator's own clone step is what already decides which repos are visible; that access-control question is deliberately kept outside this verb's trust boundary (contrast with the still-gated, API-driven scan-org, Wave 3b).

Synopsis

agenteval skills scan-workspace <path> [--format console|markdown|json] [-o|--output <file>] [--fail-on-noncompliant]
                                        [--write-baseline] [--baseline-root <dir>] [--check-baseline]
                                        [--save-manifest-baseline <file>] [--manifest-baseline <file>] [--baseline-note <text>]

What it does

Fans out scan --repo's existing per-repo pipeline over every immediate subdirectory of <path>, combining findings/coverage into one report (per-repo skill/finding counts print to stderr). Every entry's location is tagged {repoFolder}/{conventionPath}, so cross-location content drift — the same detector scan --repo already uses, unchanged — now also fires across repos, not just across conventions within one repo: the same skill name with different content in two different cloned repos is exactly the drift/poisoning signal this is for.

gh repo clone myorg/service-a ~/audit/service-a
gh repo clone myorg/service-b ~/audit/service-b
agenteval skills scan-workspace ~/audit --write-baseline --format json -o report.json

Options

Option Description
<path> Required, positional. A folder whose immediate subdirectories are repo roots.
--format <fmt> console (default), markdown, or json.
-o, --output <path> Write the rendered report to a file instead of stdout.
--fail-on-noncompliant Exit 1 when the scan finds a High-severity finding. Cross-location drift (Medium) and previously-vetted matches (Low) never trigger this — only High-severity findings do.
--write-baseline Capture a timestamped baseline snapshot across all scanned repos into the baseline ledger.
--baseline-root <dir> Baseline ledger root directory. Default .agenteval/skills-baselines-workspace — deliberately DIFFERENT from scan's .agenteval/skills-baselines default, so a plain scan --write-baseline can't accidentally get diffed against a much larger workspace-scale snapshot. Pass the same root to both verbs explicitly if you want one shared ledger.
--check-baseline Trust-on-first-use across the whole workspace — see scan's --check-baseline above.
--save-manifest-baseline <file> / --manifest-baseline <file> / --baseline-note <text> Same single-pin manifest hash-drift gate as scan — see scan's own rows above. A skill name duplicated across repos is deduplicated the same way --repo already tolerates it.

Known limitation: skills baseline diff/history track only one location per skill name even when a single snapshot legitimately has several (a pre-existing Wave 2 shortcut) — at workspace scale, where the same name across many repos is the expected case, this means the persisted ledger's diff/history can miss drift in every repo except whichever sorts first. The live scan-time CrossLocationContentDrift finding does NOT have this limitation. See Agent Skills for the full explanation.

Exit codes: same as scan above.

agenteval skills baseline

list, diff and history inspect the multi-snapshot skill baseline ledger that agenteval skills scan --write-baseline writes to. Each snapshot is a full point-in-time capture (structural fingerprint + file-content hash per skill, plus that skill's compliance findings at the time) — never overwritten, so the ledger accumulates history across scans. approve works on a different file: the single-file manifest baseline that Gatekeeper's skill check reads.

Synopsis

agenteval skills baseline list  [--baseline-root <dir>]
agenteval skills baseline diff  [--baseline-root <dir>] [--since <id>] [--skill <name>] [--hash structural|content]
agenteval skills baseline history <skill-name> [--baseline-root <dir>]
agenteval skills baseline approve <skill-name> --skill-path <dir> --baseline <file> [--note <text>]

list — every captured snapshot (Id, capture time, scanned root, skill count), oldest listed first.

diff — compares two snapshots (default: the two most recent; pass --since <id> to diff a specific snapshot against the latest) using the same ManifestDriftDetector primitive PromptTemplateDriftGate/ McpToolDescriptionPoisoningGate use, over either the structural fingerprint or the full content hash (--hash, default content — the stronger signal). A Changed skill is flagged CHANGED + NEW HIGH FINDING when the change also introduced a new High-severity compliance finding (vs. changed, no new High finding for a cosmetic-only edit) — this is the "don't cry wolf on every cosmetic edit" guard.

history <skill-name> — walks the ledger chronologically and reports every point where that skill's content hash changed, with any High-severity findings present at each change point.

approve <skill-name> — re-pins one skill in a manifest baseline file: the SkillManifestBaseline JSON that skills scan --save-manifest-baseline writes and GatekeeperOptions.SkillBaselinePath points at. Use it after reviewing a change to that skill, for example when Gatekeeper's skill check has raised a SkillDriftException. It scans --skill-path offline (no model call), finds the skill whose manifest name equals <skill-name> exactly (case-sensitive), and replaces that skill's structural fingerprint — and its content hash, when the skill has a folder on disk. Every other skill's pinned entries are left as they were. The file's capture time is set to now; --note replaces the stored note, and without it the existing note is kept. A missing file or parent directory is created.

Options

Option Description
--baseline-root <dir> (list, diff, history) Baseline ledger root directory. Default .agenteval/skills-baselines (must match what scan --write-baseline used).
--since <id> (diff only) Diff this snapshot's Id against the most recent snapshot, instead of the two most recent.
--skill <name> (diff only) Only show the diff for this skill.
--hash structural\|content (diff only) Which hash to diff. Default content.
--skill-path <dir> (approve only) Required. The skill's directory, or a parent directory holding many skills.
--baseline <file> (approve only) Required. The manifest baseline file to update.
--note <text> (approve only) A note stored on the baseline — who approved it, and why.

Exit codes: 0 on success (including "nothing to diff yet" — informational, not an error); 3 on a runtime error (e.g. --since <id> not found in the ledger, or for approve, no skill named <skill-name> under --skill-path).

agenteval compliance render

Re-render a PDF report from existing compliance evidence — no LLM cost (the evidence is already on disk).

Synopsis

agenteval compliance render --regulation <reg> --subject <name> [--ts <timestamp>] [--root <path>]
Option Description
--regulation <reg> Required. Regulation identifier: gdpr or eu-ai-act.
--subject <name> Required. Subject name to render evidence for.
--ts <timestamp> Recorded report timestamp (yyyy-MM-dd_HH-mm-ss). Defaults to most recent; an explicit timestamp can select an older captured report.
--root <path> Workspace root. Default: auto-detected.

A missing --regulation or --subject exits 2.

The reader verifies native AEF runs under .agenteval/aef/runs and reads captured rich compliance artifacts from those verified bytes. Legacy timestamp directories pass through the shared StoreV1 converter. Imported reports retain their exact bytes and are labeled imported: not sealed when it ran; an ingest seal establishes custody from conversion onward. A standalone report does not establish its original execution's timing, completion or provenance.

Missing captured content, malformed reports, conflicting identities and invalid native evidence exit 1. A legacy sibling cannot recover content omitted by the native capture policy. Selecting the latest report does not silently substitute an older captured report when a newer native report is unavailable. The bounded native scan verifies candidates before subject filtering, so an invalid or incompletely scanned native source must be resolved before rendering.

PDFs are written to .agenteval/compliance/{regulation}/{subject}/{timestamp}/report.pdf, outside the AEF source folder. Rendering leaves the source evidence and its seal unchanged. The printed integrity outcome and signature-trust status are separate; this command does not apply a signer trust policy.

agenteval render

Re-render Markdown and PDF from an original agentic report in verified AEF — no LLM cost.

Synopsis

agenteval render --benchmark <kind> --subject <name> [--ts <timestamp>] [--root <path>]
Option Description
--benchmark <kind> Required. Benchmark type (currently: agentic).
--subject <name> Required. Subject name to render results for.
--ts <timestamp> Timestamp directory. Defaults to most recent.
--root <path> Workspace root. Default: auto-detected.

A missing --benchmark or --subject exits 2.

Native runs need --aef-content on when the benchmark runs to retain the original report for later rendering. Capture off keeps the report identity but omits its document. Rendering returns 1 when captured content is unavailable or evidence is invalid or ambiguous; it does not fall back to a dual-written legacy report. Existing legacy report folders are converted on read, with ingest custody labelled separately from original execution provenance. Reports still go to .agenteval/benchmarks/agentic/<subject>/<timestamp>/.

agenteval log-file

Work with a file written by --capture-fixture.

Synopsis

agenteval log-file to-fixture <captured.jsonl> --out <fixture.json>
agenteval log-file replay     <captured.jsonl> --out <report.md> (--from-env | --endpoint <url> --model <name> [--api-key <key>]) [--strict-text]
agenteval log-file gate-replay <captured.jsonl> --baseline <gates.json> --candidate <gates.json> [--json]

to-fixture writes a JSON array of scripted turns that ScriptedChatClient.FromFixture loads, so a test can play back real model responses with no model behind it. A fixture scripts what the model answered, not what it was asked, so only responses are kept: each response line becomes a turn (text, tool calls, finish reason, token counts); each error line becomes a turn that throws the first line of the captured exception; and abandoned lines are skipped, with a note on stderr saying how many.

replay reads the capture itself — not a to-fixture output, which no longer holds the requests — and resends the request of each response line to another target, one at a time and independently: the same messages, instructions, temperature, max output tokens, model id and tool definitions. Tools are declared to the target but never run. Each round-trip is compared on structure, not wording:

Verdict When
Fail The target threw; the tool calls differ (tool names and argument names, with the number of calls); the finish reason differs; or, with --strict-text, the text differs.
Flag Tool calls and finish reason match, but the response shape differs — a length bucket (under 100, up to 500, or over 500 characters) plus whether the text has a list or a code block.
Pass Tool calls, finish reason and response shape all match.

Token usage and latency are shown for information and never change a verdict. error and abandoned lines are not replayed; the report counts them as skipped. The Markdown report is written to --out and also printed to stdout.

gate-replay answers "what would this gate change have done to real traffic?" without a model or a network call. It takes every tool call in the capture's response lines and runs two Gatekeeper tool-gate configurations over each one with the real gates (GateReplayer): --baseline, the gates in force, and --candidate, the proposed ones. It prints both verdicts for every call, marks the calls whose verdict differs, and counts the calls the candidate newly blocks and newly lets through (--json gives the same as JSON). A configuration is a JSON array of gates, named by the ids agenteval gatekeeper list-gates prints; [] is no gate:

[
  { "gate": "tool:forbidden-tool", "forbidden": ["send_email", "delete_db"] },
  { "gate": "tool:argument-pattern", "pattern": "rm\\s+-rf" },
  { "gate": "tool:domain-allowlist", "allowedDomains": ["docs.example.com"] }
]

Only these three gates, which read a call's own arguments, are replayed. A capture keeps the text of earlier messages but not tool results, so a gate that reads the conversation (tool:referential-integrity, tool:taint-tracking) would not decide here what it decided live; it is refused with that reason rather than replayed on half a history. Each gate is replayed with its default settings.

--capture-fixture masks credential shapes (keys, tokens) in what it writes, so a call whose arguments were masked does not carry what the model sent. When either configuration has an argument gate (tool:argument-pattern, tool:domain-allowlist), such a call is marked "not measured" ("measured": false in --json) and left out of the counts: its verdict would be about the mask.

Options

Option Description
<captured> Required, positional. A file written by --capture-fixture.
--out <path> Required. to-fixture: the fixture JSON array. replay: the Markdown report. A missing parent directory is created.
--from-env (replay) Replay against the provider AI_INFERENCE_PROVIDER selects (see Environment variables). --azure-from-env is its old name and still works. Checked before --endpoint.
--endpoint <url> / --model <name> / --api-key <key> (replay) Replay against an OpenAI-compatible endpoint. --model is required with --endpoint. Without --api-key, OPENAI_API_KEY is used, and without that a placeholder key for keyless local servers.
--strict-text (replay) Also fail a round-trip whose text is not identical. Off by default: model output is not reproducible, even against the same model with the same settings.
--baseline <gates.json> / --candidate <gates.json> (gate-replay) Required. The two gate configurations to compare.
--json (gate-replay) Print the comparison as JSON on stdout instead of the report.

Exit codes

Code Meaning
0 to-fixture: the fixture was written. replay: no round-trip failed (flags do not fail the run). gate-replay: the replay ran, whatever it found.
1 replay: at least one round-trip failed.
2 replay: no target was given, --endpoint was given without --model, or --from-env found no configured provider. gate-replay: a configuration file is missing, is not a JSON array of gates, or names a gate it cannot replay; or the capture holds no tool call.
3 The capture file does not exist, or another error occurred (for example, a line that is not a capture record).

agenteval mc serve

Start the Mission Control web portal — GraphQL, REST, and SPA on one port — from any working directory. Requires .NET 10. See Mission Control Getting Started.

Synopsis

agenteval mc serve [--port <N>] [--workspace <path>]
Option Env var Default Description
--port <N> (none — see note) 5000 Bind a different HTTP port. mc serve forcibly binds to http://127.0.0.1:<port> and ignores any pre-set ASPNETCORE_URLS (see Environment variables).
--workspace <path> AgentEval__Root current directory Workspace root. Mission Control reads {workspace}/.agenteval/.

The CLI spawns AgentEval.MissionControl(.exe|.dll) co-located in the same publish directory. The subprocess inherits its working directory from the CLI's bin folder so the SPA's static-asset pipeline resolves correctly; the workspace is plumbed through the AgentEval__Root env var.

Exit codes

Code Meaning
0 Stopped cleanly (Ctrl+C).
1 Port unavailable, MC assembly missing, or subprocess failed to start.
2 Running on net8/net9 — Mission Control requires .NET 10.

agenteval mc doctor

Verify Mission Control's runtime artefacts are co-located with the CLI and the SPA bundle is intact. Useful diagnostic before mc serve fails with a less-informative error. Sibling to agenteval doctor (which validates workspace data, not portal binaries). Requires .NET 10.

Synopsis

agenteval mc doctor

What it checks

  1. AgentEval.MissionControl.dll (and .exe on Windows) is present alongside the CLI.
  2. wwwroot/ exists with index.html and a populated assets/ folder (JS + CSS bundles).
  3. The Web SDK's static-asset manifest (*.staticwebassets.endpoints.json or *.runtime.json) is present.
  4. On non-Windows, dotnet is on PATH (the CLI spawns the MC .dll via dotnet).

Prints Errors: N | Warnings: N | OK: N and exits 2 on any error.

agenteval assert-ai

Work with Microsoft's ASSERT (assert-ai 0.3.0 formats). The guide is ASSERT Interoperability.

Synopsis

agenteval assert-ai serve (--from-env [--model <m>] | --endpoint <url> --model <m> [--api-key <key>]) [--system-prompt <text>] [--port 8765] [--path /assert] [--host localhost]
agenteval assert-ai import <run-dir> [--format text|json|markdown] [-o <file>] [--taxonomy <file>] [--test-set <file>] [--calibration <file>] [--max-harm-rate <0-1>] [--max-over-refusal-rate <0-1>] [--max-unmeasured <0-1>]
agenteval assert-ai export --golden <file.jsonl>... --taxonomy <taxonomy.json> --judge-model <model> --out <dir> [--evaluator <key>...] [--suite agenteval] [--run judge-1] [--assert-root <path>]
agenteval assert-ai calibrate <run-dir> --cases <agenteval-cases.json> [--format text|json] [-o <calibration.json>]

serve answers ASSERT's endpoint requests ({message, history} → {response, events}) with the model your provider variables select, or an OpenAI-compatible endpoint, until Ctrl+C. Point ASSERT's pipeline.inference.target.endpoint at the URL it prints; keep the host name localhost (ASSERT refuses a literal 127.0.0.1 unless ASSERT_ALLOW_PRIVATE_ENDPOINTS=1). --host + answers every host name, for ASSERT in a container calling host.docker.internal; on Windows that needs a URL reservation (netsh http add urlacl url=http://+:8765/ user=Everyone, as administrator). It serves a bare model; to serve an agent with tools, use AssertAiTarget from code.

import reads an ASSERT run directory (the one holding scores.jsonl) and prints ASSERT's harm and over-refusal rates per kind of test case, computed as ASSERT computes them, every case's verdict, and the cases with no score row. A failed judge is an error and a case with no row is named, never counted as a pass. Exit 1 when a rate is above --max-harm-rate or --max-over-refusal-rate; 11 when a gate is set and it cannot pass: more cases unmeasured than --max-unmeasured allows (default none), the run's manifest says it did not complete, or no rate could be measured; 2 for a directory or file that cannot be read.

export writes labelled golden cases (the agentic calibration JSONL format) as an ASSERT judge-only run, with the config to run it and agenteval-cases.json mapping ASSERT's positional ids back to the case ids. --taxonomy is required: ASSERT's judge cannot run without one.

calibrate compares ASSERT's verdicts on exported cases with their labels: accuracy with a 95% Wilson interval, Cohen's κ, dangerous errors (a labelled failure not flagged, including when the judge found no category relevant) and false alarms; a case the judge did not decide is counted as not measured. It refuses a run whose transcripts are not the ones exported with the case map. -o writes the result for import --calibration, which attaches it only to runs of the same judge on the same taxonomy. Exit 11 when the judge decided no case.

agenteval aef

Work with AEF 1.0 runs, the AgentEval Evidence Format (a release candidate). The guide is AEF Evidence Format.

Synopsis

agenteval aef verify <run-dir> [--policy <trust-policy.json>] [--anchors <run-hashes.json>] [--json]
agenteval aef seal <run-dir> [--key <pkcs8.pem>] [--sealed-by producer|ingest] [--json]
agenteval aef view <run-dir> [--at <time>] [--policy <trust-policy.json>] [--json]
agenteval aef checkpoint <manifest.json> --runs <dir> [--policy <trust-policy.json>] [--at <time>] [--envelope <file>] [--json]
agenteval aef export <store-dir> <out-dir> [--run <run-id>] [--target-mode live|replayed|scripted|mocked] [--content-capture on|off] [--no-seal] [--key <pkcs8.pem>] [--json]
agenteval aef import store-v1 <store-dir> <out-dir> [--run <run-id>] [--target-mode live|replayed|scripted|mocked] [--content-capture on|off] [--no-seal] [--key <pkcs8.pem>] [--json]
agenteval aef import assert-ai <assert-run-dir> <out-dir> [--taxonomy <file>] [--test-set <file>] [--calibration <file>] [--max-harm-rate <0-1>] [--max-over-refusal-rate <0-1>] [--content-capture on|off] [--no-seal] [--key <pkcs8.pem>] [--json]
agenteval aef import otel <logs-file> <out-dir> --run-id <id> --from <tool> --subject <kind:name> [--subject-kind agent] --target-mode live|replayed|scripted|mocked [--content-capture on|off] [--no-seal] [--key <pkcs8.pem>] [--json]
agenteval aef export-otel <run-dir> <out-file> [--policy <trust-policy.json>] [--json]
agenteval aef import inspect <log-file> <out-dir> --target-mode live|replayed|scripted|mocked [--content-capture on|off] [--no-seal] [--key <pkcs8.pem>] [--json]
agenteval aef export-inspect <run-dir> <out-file> [--ignore-overlays] [--policy <trust-policy.json>] [--json]

import store-v1 converts an existing legacy run folder, or a .agenteval workspace selected with --run, through the same converter as the retained export spelling. Its report says imported: not sealed when it ran; an ingest seal protects the conversion from import onward, not the original execution's history. Source files are preserved. See legacy import for the source-folder contract.

An explicit workspace run folder is imported exactly as selected. A workspace plus --run must identify one folder; duplicate IDs across subjects are refused as ambiguous. Matching compliance reports must agree on subject, source run ID and the recorded source hash. With --content-capture on, rich report bytes are retained and cited by the corresponding scenarios; off retains only generic compliance evidence. Agreement with a recorded hash does not establish the original execution's custody.

verify reports a run as intact, unsealed or invalid, with every problem. With --policy, it also says which trusted identities signed it, and lets authorized redactions withhold blobs; with --anchors, whether one of the run hashes you trust is the run's. Exit 1 when the run is invalid.

seal writes seal.json for a closed run, and attestation.dsse.json when --key gives a signing key (ECDSA P-256). --sealed-by producer (the default) is for the run's own producer, which may seal only a run with no problem; ingest is for a host taking custody of a run. Exit 1 when the run cannot be sealed.

view shows the run as its overlays leave it at --at (default: now): overridden results, reviews, waivers in force, withheld blobs. The sealed files never change.

checkpoint checks a checkpoint manifest alone, against the runs it names (found under --runs by their run.json), each lane's recomputed result and its decision, and its signature (<name>.dsse.json beside the manifest, or --envelope). Exit 1 on any problem.

export converts a run of AgentEval's .agenteval/ store (store v1) into an AEF run marked as imported. Give a run folder, or the workspace with --run. Store v1 does not record how the target was driven, so --target-mode defaults to mocked: an exported run is never passed off as live evidence unless you say it was. The run is sealed as ingest unless --no-seal.

import assert-ai converts an ASSERT run into an AEF run: a root line and harm and over-refusal lines per case, ASSERT's rates in summary.json, with optional gate limits. With --content-capture off, no prompt, response, transcript or judge reasoning is kept.

import otel converts OpenTelemetry gen_ai.evaluation.result events (OTLP/JSON logs, one LogsData per line) into an AEF run, as the OpenTelemetry interop page maps them: a line per event, the run's id, subject and target mode from you, contentCapture on unless --content-capture off (then logs carrying content are refused), sealed as ingest unless --no-seal, and verified. Events the page refuses (two services, an event with no case or no name, a second event of one case with one name, …) exit 2, naming the rule.

export-otel writes an AEF run as OpenTelemetry events: one LogsData line per result line, one gen_ai.evaluation.result event per score, labelled with the result's sealed state (overlays are not applied) and parented to its traceLink. The run must verify (intact or unsealed; --policy lets an authorized redaction withhold a reasoning blob); the output file must not exist. A run it cannot export (one that does not verify, a time OpenTelemetry cannot hold, a reasoning blob that is not UTF-8 or is over 4 MiB) exits 2, and nothing is written.

import inspect converts an Inspect eval log in .json form (for an .eval log, run inspect log dump first) into an AEF run, as the Inspect interop page maps it: a line per sample and score (trial = epoch − 1 when the log has more than one epoch), a rollup line per reduction, the summary recomputed from the lines in one lane main, and the run's usage from stats. The subject is model:<eval.model>; the target mode is yours, and contentCapture is on unless --content-capture off (then no case content, explanation or answer is kept). A closed log is sealed as ingest unless --no-seal; a started log gives a running run, which is not sealed. The log is read as I-JSON within a nesting depth of 64, Inspect's bare NaN allowed only where an unscored value can be. A log the page refuses (one that is not I-JSON, such as a member named twice, Infinity or a number that overflows; a time without an offset; a numeric id that is no integer of at most 2^53 − 1; a boolean value or a string other than C, I, P and N; a reducer without an AEF value; a missing reduction; a mean the lines do not give; Score.history, an invalidation or log_updates; …) exits 2, naming the rule, and nothing is written.

export-inspect writes an AEF run as one Inspect EvalLog in .json form: a sample per case and trial, a score per result line under its path (the AEF facts Inspect has no field for under metadata.aef), a reduction per rollup line, an EvalScore per summary entry; eval.model is the subject's ref with its name decoded. A run with contentCapture: off gives no explanation, not even a line's reason. The run must verify (intact or unsealed; --policy lets an authorized redaction withhold a blob, which is then left out), and its sealed lines are exported: a run with overlay events is refused unless --ignore-overlays leaves them out. A run the page refuses (no suite, two judges, two summary entries at one path, a count metric, output or transcript evidence a line cites, a reasoning or content blob that is not UTF-8, …) exits 2, naming the rule, and nothing is written.

Every verb takes --json and then prints one JSON value. Exit 2 is a usage or input error.

Exit codes

The CLI's exit-code contract, so CI can branch on the outcome. Source of truth: src/AgentEval.Cli/ExitCodes.cs.

Code Meaning
0 Success — passed / allowed / no gate blocked.
1 Test failure — one or more evaluations failed (eval, redteam).
2 Usage error (bad flags, malformed input). Reserved strictly for bad-argument paths — see BUG-22 below. Every command returns it for a parse error (see below). bench, compliance render and render also return it for an argument they reject themselves before the run starts: a missing --subject; an unknown preset or domain pack (every bench family) or vertical (bench typedmemeval); an invalid --budget-tier; an invalid --sut configuration or --endpoint without --model; a --response-file that cannot be read; no target at all (every bench family that grades an agent, and redteam and eval), --sut mock together with a real target, or a --response without the --input it answered. A missing .agenteval/ workspace is not an argument error: those commands exit 1 and tell you to run agenteval init-workspace.
3 Runtime error (connection/model/IO failure). Also returned when a judge fails to build (JudgeFactory — missing or partial Azure OpenAI credentials, or a thrown exception constructing the client): that's a runtime/config problem, not a bad CLI argument. redteam --sut gatekeeper-demo --baseline also returns it when the baseline was taken on a different model (scripted vs real, or another real model).
4 Regression vs a supplied --baseline — redteam --fail-on regression gate (a NEW finding vs pre-existing).
5 gatekeeper inspect — a gate Blocked on real evidence.
6 gatekeeper inspect — fail-closed: the CLI could not evaluate (e.g. a history gate with no messages). Not a policy block.
7 gatekeeper inspect — not certified: the honesty guard refused an un-calibrated judge (run calibrate --certify, or pass --allow-uncalibrated).
8 redteam --sut copilot-studio — a live scan hit its --max-credits cap (BudgetExceeded). Enforced as an ESTIMATE (turns counted, not metered spend — the SDK exposes no real credit-cost field); see Copilot Studio.
9 bench <family> / bench <reg> calibrate — the composite/calibration gate is a hard FAIL (for calibration: a pillar or category FAIL, INFRA-FAIL or INCOMPLETE).
10 bench <family> — the composite gate is a WARN (soft finding, below ideal but not a hard failure). Calibration commands never return this — their thresholds are pass/fail binary.
11 No conclusive verdict, and nothing failed. bench <family> — the composite gate could not produce a conclusive verdict (e.g. skipped). Every --sut mock run exits 11 whatever it scores: it measured no agent. eval --runs N — a test case's verdict is undecided or not measured. eval --metrics … — a requested metric was not measured on some case. bench <reg> calibrate / bench agentic calibrate — a pillar or category is NOT MEASURED (nothing scored). A failure still exits 1 (eval) or 9 (calibrate).
12 Reserved by ADR-031. No command returns it.
13 compare — incomparable: the two runs could not be shown to have measured the same thing, so no delta was emitted (Incomparable). Distinct from 11, which means a run produced nothing scoreable: here both runs produced verdicts, and comparing them would be meaningless. See agenteval compare. redteam --baseline — the baseline or the current scan has no probe that reached a verdict, so the comparison is refused, neither a regression (4) nor a pass; see Red Team Security.

Parse errors exit 2. An unknown command or option, a value that does not parse (--runs abc), a missing required option, or no command at all exits 2. System.CommandLine prints the errors to stderr and the command's help to stdout, as it always has; only the exit code differs from System.CommandLine's default, which is 1 and would read as a test failure. --help exits 0, including on an otherwise incomplete command line (agenteval eval --help without --dataset), and so does agenteval --version.

redteam uses 1 for failure, 3 for runtime error, and 4 for a --fail-on regression gate. Code 8 is returned by a live --sut copilot-studio scan that hits --max-credits (BudgetExceeded) — an estimate, not a metered value. gatekeeper's 5/6/7 are deliberately distinct exit codes — see Gatekeeper from any language.

BUG-22 (resolved 2026-07-19): code 2 used to be overloaded — bench/calibrate returned it for gate FAIL/WARN as well as bad arguments, and JudgeFactory config failures also returned it, so CI could not tell "invoked wrong" from "agent failed the gate" from "judge misconfigured". This is now split across 2 (bad arguments only), 3 (judge/runtime config problems), and 9/10/11 (gate outcomes) as documented above. This is a breaking change for any external CI pipeline that branched on exit code 2 from bench/ calibrate commands — update those pipelines to check the new codes. See src/AgentEval.Cli/ExitCodes.cs and CHANGELOG.md.

See Also

Edit this page