Pressure Chamber

1. Introduction

What is Pressure Chamber?

Pressure Chamber is a systematic benchmarking framework for AI agents that measures real business outcomes under controlled resource constraints. Instead of evaluating agents on synthetic tasks with unlimited resources, we test them in conditions that mirror production environments: limited tokens, capped API calls, strict time budgets, and restricted tool access.

The core insight: an agent's true capability isn't revealed when resources are abundant — it's revealed when resources are scarce.

Why Synthetic Benchmarks Fail

Standard LLM benchmarks measure the wrong things. They tell you an agent can solve math problems or pass bar exams — not whether it can generate qualified leads or catch revenue anomalies before they cost you money.

The Case for Business-Grounded Metrics

Pressure Chamber inverts the benchmark paradigm. Instead of measuring capability in isolation, we measure value delivered per resource consumed. The framework answers three questions:

1. Does the agent accomplish business objectives? Not "can it reason" but "did it find leads, write converting copy, catch the anomaly?"
2. At what cost? Tokens consumed, API calls made, wall-clock time elapsed, tools invoked.
3. How does it degrade? When we reduce resources by 50%, does outcome quality drop 10% or 90%?

This tells operators something benchmarks never could: which agent delivers the best ROI for their specific workload under their specific resource constraints.

2. Constraint Dimensions

Pressure Chamber tests agents across five independent constraint axes. Each axis can be varied from "abundant" (typical benchmark conditions) to "scarce" (production-realistic conditions) to "extreme" (stress testing).

Constraint Interaction Matrix

Constraints don't operate in isolation. The real test is how agents handle multiple simultaneous constraints. We test standard constraint combinations:

3. Measurement Framework

Primary Metrics (Business Outcomes)

These are the metrics that matter. Everything else is instrumental.

Secondary Metrics (Operational)

Secondary metrics explain why primary metrics are what they are. They're diagnostic, not evaluative.

Normalization Across Task Types

Different tasks have different value scales. A lead gen task producing 50 leads isn't comparable to an analytics task producing 1 report. We normalize using task-specific value functions:

# Normalization Functions

def normalize_lead_gen(raw_leads: list) -> float:
    """Convert lead count to normalized 0-100 score."""
    qualified = sum(1 for l in raw_leads if l.qualification_score > 0.7)
    weighted = sum(l.estimated_value for l in raw_leads)
    return min(100, (qualified * 10) + (weighted / 1000))

def normalize_analytics(report: Report) -> float:
    """Score report on completeness, accuracy, actionability."""
    completeness = len(report.sections) / expected_sections
    accuracy = report.validated_claims / report.total_claims
    actionability = len(report.recommendations) * 5
    return min(100, (completeness * 30) + (accuracy * 40) + actionability)

def normalize_content(content: Content) -> float:
    """Score content on quality metrics."""
    seo_score = content.keyword_density_score * 20
    readability = content.flesch_score / 2
    engagement = content.predicted_ctr * 100
    return min(100, seo_score + readability + engagement)

Composite Scoring

The final Pressure Chamber score combines primary and secondary metrics into a single comparable value:

Where:

• Outcome_Norm: Normalized primary metric (0-100)
• Efficiency: 100 - (cost_percentile + latency_percentile) / 2
• Reliability: 100 × (1 - error_rate)

4. Task Categories

Pressure Chamber evaluates agents across five core business task categories. Each category has standardized task variants with known baseline performance.

Lead Generation

Find, qualify, and prepare outreach for potential customers.

Analytics & Reporting

Transform raw data into insights and formatted reports.

Content Creation

Generate marketing and sales content optimized for specific channels.

Outreach Automation

Execute communication sequences across channels.

Anomaly Detection

Identify unusual patterns requiring human attention.

5. Testing Protocol

Baseline Establishment

Before constraint testing, we establish each agent's maximum capability:

1. Run each task 10 times under Baseline (abundant) conditions
2. Record all metrics for each run (outcome, cost, latency, errors)
3. Calculate statistics: mean, standard deviation, min/max for each metric
4. Establish baseline score = mean outcome with P90 latency and cost

# Baseline Protocol

for task in TASK_CATALOG:
    results = []
    for run in range(10):
        result = execute_task(
            agent=agent,
            task=task,
            constraints=BASELINE_PROFILE  # Abundant resources
        )
        results.append(result)
    
    baseline = {
        'outcome_mean': mean([r.outcome for r in results]),
        'outcome_std': std([r.outcome for r in results]),
        'latency_p90': percentile([r.latency for r in results], 90),
        'cost_mean': mean([r.cost for r in results]),
        'error_rate': sum([r.error for r in results]) / 10
    }
    save_baseline(agent, task, baseline)

Constraint Variation Methodology

Once baselines are established, we systematically vary constraints:

Single-Axis Variation

Test each constraint axis independently while holding others at Production level:

# Single-Axis Testing

AXES = ['tokens', 'api_calls', 'time', 'tools', 'noise']
LEVELS = ['abundant', 'normal', 'constrained', 'extreme']

for axis in AXES:
    for level in LEVELS:
        constraints = PRODUCTION_PROFILE.copy()
        constraints[axis] = LEVEL_VALUES[axis][level]
        
        for run in range(5):
            result = execute_task(agent, task, constraints)
            record_result(axis, level, result)

Multi-Axis Profiles

Test standard constraint combinations that represent real deployment scenarios:

• Production Profile: 5 runs — typical operating conditions
• Budget Profile: 5 runs — cost-constrained environments
• Stress Profile: 3 runs — failure mode discovery

Statistical Significance Requirements

We require statistical rigor before drawing conclusions:

from scipy import stats

def compare_agents(agent_a_results, agent_b_results):
    """Determine if Agent A outperforms Agent B with significance."""
    
    # Welch's t-test (unequal variances)
    t_stat, p_value = stats.ttest_ind(
        agent_a_results, 
        agent_b_results, 
        equal_var=False
    )
    
    # Effect size (Cohen's d)
    pooled_std = np.sqrt(
        (np.var(agent_a_results) + np.var(agent_b_results)) / 2
    )
    cohens_d = (
        np.mean(agent_a_results) - np.mean(agent_b_results)
    ) / pooled_std
    
    significant = (p_value < 0.05) and (abs(cohens_d) > 0.5)
    
    return {
        'p_value': p_value,
        'cohens_d': cohens_d,
        'significant': significant,
        'winner': 'A' if cohens_d > 0 else 'B' if significant else 'tie'
    }

Reproducibility Standards

All Pressure Chamber evaluations must be reproducible. Requirements:

1. Seed Locking: All random seeds recorded and reproducible
2. Version Pinning: Exact model versions, not aliases (e.g., gpt-4-0125-preview not gpt-4)
3. Prompt Hashing: SHA-256 hash of all prompts stored with results
4. Tool Versioning: Tool implementations versioned and archived
5. Environment Capture: Full dependency snapshot for each evaluation run

# Reproducibility Manifest

{
  "evaluation_id": "pc_2026-03-02_a4f2c1",
  "agent": {
    "name": "kai-cmo-v3",
    "model": "claude-3-5-sonnet-20250301",
    "prompt_hash": "sha256:8f14e45f..."
  },
  "environment": {
    "python": "3.11.8",
    "dependencies_lock": "sha256:2c26b46b...",
    "random_seed": 42
  },
  "tasks": {
    "lead_gen_company_research": {
      "version": "1.2.0",
      "input_hash": "sha256:fcde2b2e..."
    }
  },
  "results_hash": "sha256:7d865e95..."
}

6. Interpretation Guidelines

How to Read Results

The Degradation Curve

The most important output is the degradation curve: how does outcome quality change as constraints tighten?

Graceful degradation (linear or log curve) indicates robust architecture. The agent makes intelligent tradeoffs as resources shrink.

Cliff-edge degradation (sudden collapse) indicates fragile design. The agent depends on a specific resource threshold and fails catastrophically below it.

Efficiency Frontier

Plot agents on Outcome vs. Cost axes. The efficiency frontier shows which agents deliver the best outcomes at each cost level:

Agents on the frontier are optimal choices — none dominate them on both axes. Agents below the frontier are strictly worse than a frontier agent.

Common Pitfalls

When Constraints Help vs. Hurt

Counterintuitively, constraints sometimes improve outcomes:

Reporting Template

Standard Pressure Chamber reports include:

# Pressure Chamber Report: [Agent Name]

## Executive Summary
- Overall PC Score: 78.4 (Production profile)
- Efficiency Rank: 2nd of 5 tested agents
- Recommendation: Deploy for lead gen; avoid for analytics

## Baseline Performance
| Task Category    | Outcome | Latency P90 | Cost Mean |
|------------------|---------|-------------|-----------|
| Lead Generation  | 92.1    | 45s         | $2.30     |
| Analytics        | 71.3    | 120s        | $4.10     |
| Content Creation | 88.7    | 90s         | $3.50     |

## Degradation Analysis
- Token constraints: Graceful (−15% at Constrained)
- Time constraints: Cliff-edge below 30s (−65%)
- Tool constraints: Graceful (−20% at Minimal)

## Constraint Sensitivity
Most sensitive: Time (drives 40% of variance)
Least sensitive: Noise (only 8% impact at 40% noise)

## Recommendation
Strong choice for time-flexible lead gen tasks.
Avoid for latency-critical or analytics workloads.

Version History