AI Agent Evaluation & Regression Complete Package

Project Plan A: Deterministic Agent Mock & Replay Harness (VCR for AI Agents)

Overview

A lightweight, high-performance recording and replay harness designed to eliminate non-determinism, third-party API costs, and network latency during AI agent CI/CD testing. Inspired by VCR-style HTTP mocking, this tool intercepts tool calls and external API interactions during an initial agent run, serializing them into deterministic test fixtures for rapid offline regression testing.

Key Technical Objectives

• Deterministic Fixture Generation: Capture outbound JSON-RPC/REST calls, arguments, and return states into versioned, human-readable .yaml or .json trace files.
• Zero-Cost CI Runs: Intercept outbound tool calls during automated test suites and mock responses directly from local fixtures, bypassing expensive LLM calls and third-party APIs.
• Fuzzy & Exact Payload Matching: Support exact argument matching as well as configurable schema-based fuzzy matching for dynamic parameters (e.g., timestamps, generated UUIDs).
• State Drift & Break Detection: Instantly catch breaking API schema changes, invalid argument mutations, and infinite tool-execution loops before code hits production.

System Architecture & Workflow

[ Agent Under Test ] 
        |
        v  (Outbound Tool Request)
+-----------------------------------------------------------+
|                   Mock & Replay Proxy                     |
|                                                           |
|  Mode: RECORD  ---> Save request/response to local trace  |
|  Mode: REPLAY  ---> Match against fixture & inject response|
+-----------------------------------------------------------+
        |                                    |
        v (RECORD Mode)                      v (REPLAY Mode)
[ Live External API ]              [ Saved Local Fixtures ]
      

Core Components

1. Interceptor Proxy Engine: Middleware interceptor written in Go or Python that acts as a local gateway for outbound agent tool execution.
2. Fixture Serialization Engine: Converts execution traces into structured, version-controlled mock files containing sequence indexes and payload hashes.
3. Sequence Matcher: Ensures tool invocations happen in the exact expected sequence, raising immediate regression flags if an agent skips steps or changes tool call order.

Deliverables & Milestones

• Phase 1: Core HTTP/RPC proxy interceptor and JSON fixture recorder.
• Phase 2: Sequence-matching engine with configurable tolerance for dynamic fields (UUIDs, timestamps).
• Phase 3: Integration runner for PyTest / Go testing framework.
• Phase 4: Benchmarking suite demonstrating 10x faster test execution times and 100% cost reduction in offline CI environments.

Project Plan B: Dual-Layer Automated Agent Regression & Evaluation Suite

Overview

An end-to-end evaluation pipeline built to detect performance, accuracy, and behavioral regressions in multi-step AI agents. By combining fast, low-cost deterministic assertions with an automated LLM-as-a-Judge semantic evaluator, this suite provides comprehensive pass/fail metrics across model fine-tunes, prompt revisions, and framework updates.

Key Technical Objectives

• Multi-Layer Regression Gating: Enforce a two-tier evaluation strategy—Layer 1 for zero-cost structural/schema validation, and Layer 2 for semantic quality scoring.
• Trajectory Divergence Analysis: Trace and compare candidate execution pathways against established baseline trajectories to detect extra steps, hallucinated tools, or plan degradation.
• Automated LLM-as-a-Judge Scoring: Standardize qualitative checks (e.g., safety, conciseness, tone, adherence to instructions) using structured evaluation rubrics and automated scoring models.
• CI/CD Pass/Fail Thresholds: Output structured JSON/JUnit test results with customizable threshold metrics to automatically block regressive PRs in automated build pipelines.

Evaluation Layer Architecture

Core Components

1. Trajectory Collector: Captures full execution trees, including thoughts, tool inputs, outputs, execution latencies, and final system states.
2. Deterministic Assert Engine: Evaluates tool call sequence accuracy, schema validity, and max-step limits in milliseconds.
3. Semantic Rubric Judge: Sends execution traces and rubrics to an evaluation model, outputting normalized scores (0.0–1.0) with detailed reasoning logs.
4. Diff & Reporting Generator: Produces side-by-side terminal and HTML diff reports comparing baseline vs. candidate trajectories.

Deliverables & Milestones

• Phase 1: Trajectory schema specification and log collector implementation.
• Phase 2: Deterministic assertion library (tool order validation, schema checks, max-step bounds).
• Phase 3: Structured LLM-as-a-Judge grading harness with rubric prompt templates and JSON output parsing.
• Phase 4: GitHub Actions integration showcasing automated pass/fail PR checks with visual diff artifacts.

OneHack Community Discourse Post

Here is the complete forum post formatted for OneHack to start a discussion about agent evaluation and regression testing:

Title: Stop Guessing: Why We Need Real Regression Testing for Our AI Agents (And How to Build It)

Hey OneHack Community,

We're all building and experimenting with AI agents—whether it's custom LangChain tools, AutoGPT setups, or custom Go/Python agent runtimes using models like Gemini, Claude, or Llama.

But let’s talk about a dirty secret in agent dev: **How do you actually know your agent isn't getting worse over time?**

You update a prompt, bump an API version, tweak a tool schema, or switch base models to save money... and suddenly your agent starts:
1. Getting stuck in infinite tool loops.
2. Hallucinating API arguments that don't exist in your schema.
3. Taking 6 unnecessary steps to answer a simple query.
4. Silently failing on edge cases you thought you fixed weeks ago.

If you're relying on "vibe checks" (typing 2 prompts into a playground and saying "yep, looks good"), your agent *will* break in production.

### How We Fix It: Deterministic VCR Mocks + Dual-Layer Evaluation

I've been designing a blueprint for an open-source **Agent Regression Framework** that treats agent testing like real software engineering.

Here are the two core ideas that change the game:

#### 1. "VCR-Style" Tool Replay (Zero-Cost Offline CI)
Instead of paying API fees every time your CI pipeline runs tests, you intercept tool/API calls on the first run and record them to a local JSON trace. On future runs, your test runner replays those responses offline. 
- Fast (runs in milliseconds)
- Free (no LLM/tool API charges)
- Deterministic (catches schema breaks instantly)

#### 2. Dual-Layer Gating
Don't waste expensive evaluation calls on broken code.
- **Layer 1 (Deterministic Gate):** Check step-count limits, OpenAPI JSON schema validity, and tool DAG order using standard code assertions.
- **Layer 2 (Semantic Gate):** Use embedding cosine similarity or an "LLM-as-a-Judge" rubric *only* if Layer 1 passes to verify qualitative answer accuracy.

---

### What are you guys using to monitor your agents?

Are you running automated evaluation pipelines, or are you still relying on manual testing? Let's discuss—what's the weirdest way one of your agents has regressed or gone off the rails when you tweaked its prompts?

I'm working on publishing open-source blueprints and test harnesses for this. Drop your thoughts below!