Concepts
Traces in
Every ingestor produces the same TraceEvent: messages, output, model,
tool calls, usage, status. Supported inputs: LiteLLM standard logging payloads
(or the minimal {messages, response, model} shape), OpenTelemetry GenAI
spans (flattened or blob attributes), and sec-MCP v1 tool-call logs. Redaction
profiles (default, bharat, dpdp-strict) replace
PII with deterministic pseudonyms so clustering stays stable.
sec-MCP contract
sec-MCP tool-call logs follow the versioned
schemas/secmcp-log-v1.json contract: v, tool,
arguments, an allow|deny policy decision with probability, latency,
and an optional LLM block. swapgate capture --from secmcp consumes
exactly this shape; tool-only records become eval inputs with the decision
preserved on the event.
Evals out
Events cluster offline by template signature plus shingle similarity — no embeddings,
no network. Each cluster yields eval cases with heuristic rubrics
(contains, regex, length bounds, JSON keys). Nothing runs
until a human reviews the pending suite and approves it. Synthesised rubrics are a
starting draft, never ground truth.
Judges
| Judge | How it scores | When to use it |
|---|---|---|
deterministic | Every rubric check must pass; score is the fraction passed. | CI gates. Fast, explainable, reproducible. |
typed | Deterministic signals mapped to pass / fail / needs-review with a probability. | Borderline triage; pluggable backends later. |
llm | An OpenAI-compatible judge model returns one-line PASS/FAIL verdicts. | Rubrics humans can state but regex cannot check. |
The regression rule
Both providers replay every case and the judge scores both. The candidate
FAILS when its pass rate drops below
--fail-under (absolute floor) or more than --max-regression
below the baseline (relative rule). Either breach exits 2.
Receipts
Each run emits a decision receipt: suite identity, both pass rates, per-case outputs
and verdicts, a sha256 content hash, and an HMAC-SHA256 signature under a local key.
swapgate verify re-checks both. Tamper-evident and offline; Ed25519 is the
documented path to third-party verification. The TypeScript SDK verifies receipts in Node.
MCP server
swapgate serve-mcp exposes compare_models and
list_evals over stdio JSON-RPC, so coding agents can run the gate themselves.