How Do You Know Your AI Actually Works? Build an Evaluation Harness

One of the easiest mistakes to make with an AI application is to confuse a convincing demonstration with a reliable system.

A prompt works on ten examples. The output looks good. Someone in the team says, “That’s surprisingly accurate.”

And then the prompt changes.

Or the model changes.

Or the documents being retrieved change.

Or a customer asks something slightly different.

Suddenly nobody can answer a basic engineering question:

Did we make the system better or worse?

I wanted a simple, repeatable way to answer that question.

So this experiment is about building an evaluation harness for an AI application — something small enough to build in an afternoon, but useful enough to become part of the development workflow.

The goal isn’t to create a perfect benchmark.

The goal is to create a feedback loop.

Change something → run the same tests → measure the result → understand the failures → decide what to do next.

The experiment

Take an AI task that matters to your application and create a fixed set of representative test cases.

For each case, record the input, expected behaviour, actual AI response, whether the response passed, latency, token or usage cost where available, and the failure category.

Then run the same cases every time you change something important.

For example, imagine an AI assistant that classifies incoming support requests.

A test case might look like:

{
  "input": "I can't log in because my account is locked",
  "expected_category": "ACCOUNT_ACCESS",
  "expected_priority": "HIGH"
}

The AI might return:

{
  "category": "ACCOUNT_ACCESS",
  "priority": "HIGH"
}

That’s a pass.

Now change the prompt.

Run the same case again.

If it suddenly returns GENERAL_SUPPORT, you have discovered a regression before your users do.

What we’re trying to measure

Don’t start with dozens of metrics.

For the first experiment, I’d use five:

1. Task accuracy

How often did the system produce the expected result?

accuracy = passing cases / total cases

2. Failure modes

A single accuracy number hides useful information.

Classify failures where possible:

• incorrect classification
• missing information
• unsupported claim
• malformed output
• refusal when an answer was expected
• answer when the system should have refused
• retrieval failure

The interesting engineering question is often why the system failed, not simply how often.

3. Latency

Record at least average latency and p95 latency.

AI applications can feel fine during development and become painful when the system has to process real workloads.

4. Cost

Record the approximate cost per test case.

A change that increases accuracy from 91% to 92% but doubles the cost may be useful — or it may not.

The harness gives you the evidence to make that decision.

5. Regression rate

This is the metric that makes the harness useful over time.

Suppose version A passes 94 of 100 cases.

You change the prompt and version B passes 95.

That’s encouraging.

But suppose the one additional pass came at the expense of five cases that previously worked.

Your overall result has actually deteriorated.

A useful evaluation therefore compares versions, rather than looking only at the latest result.

Build your first evaluation dataset

Start with 50–100 cases, not 10,000.

You want representative examples rather than a huge dataset.

A good starting distribution might be:

• 60% ordinary cases
• 20% difficult or ambiguous cases
• 10% edge cases
• 10% known failure cases

If you already have production examples, sample from them after removing sensitive information.

If you don’t have production data yet, create the cases deliberately.

The important thing is to document where they came from.

Step 1: Define the task

Pick one narrow AI capability.

Good candidates include:

• classification
• extraction
• summarisation
• question answering
• routing
• document analysis
• structured decision support

Avoid evaluating an entire AI product initially.

You want one question that can be answered clearly.

For example:

Can this model correctly classify customer support requests into the categories our service desk uses?

That’s much easier to measure than:

Is our AI assistant good?

Step 2: Create the test cases

Put your cases in a simple JSON or CSV file.

For example:

[
  {
    "id": "001",
    "input": "My account is locked",
    "expected": {
      "category": "ACCOUNT_ACCESS",
      "priority": "HIGH"
    }
  },
  {
    "id": "002",
    "input": "How do I change my email address?",
    "expected": {
      "category": "ACCOUNT_SETTINGS",
      "priority": "LOW"
    }
  }
]

Keep the dataset outside your prompt.

That distinction matters.

The application should consume the test cases in exactly the same way it consumes normal requests.

Step 3: Establish a baseline

Before experimenting with different models or elaborate prompting techniques, run your current implementation against the dataset.

Record the result.

For example:

Metric	Baseline
Test cases	100
Passed	87
Accuracy	87%
p95 latency	1.8s
Cost / case	£0.003
Malformed responses	2

These numbers are illustrative.

The important thing is that your experiment should produce your own numbers.

Don’t manufacture a benchmark simply because a benchmark looks impressive.

Step 4: Capture the complete response

For every test case, store enough information to understand what happened.

A useful record looks something like:

{
  "case_id": "001",
  "model": "your-model",
  "prompt_version": "v3",
  "passed": true,
  "latency_ms": 1240,
  "input_tokens": 84,
  "output_tokens": 31,
  "response": {
    "category": "ACCOUNT_ACCESS",
    "priority": "HIGH"
  }
}

You don’t need a sophisticated observability platform for the first experiment.

A JSON file or SQLite database is enough.

Step 5: Make evaluation deterministic where possible

AI outputs are probabilistic.

Your evaluation shouldn’t be.

For structured tasks, make the output schema explicit.

For example:

{
  "category": "ACCOUNT_ACCESS",
  "priority": "HIGH"
}

Then validate the response before evaluating it.

A malformed response should not accidentally count as a successful answer because a human could understand what the model meant.

That’s an important engineering distinction:

human-readable is not necessarily machine-reliable.

Step 6: Run the baseline repeatedly

If your system uses a stochastic model configuration, run the dataset more than once.

You may discover that the same test case passes on one run and fails on another.

That is itself a result.

You can then start measuring not only:

Does it work?

but:

How consistently does it work?

For example, run 100 cases three times and calculate the pass rate for each run.

You might find:

Run 1: 91%
Run 2: 94%
Run 3: 92%

That tells you something important that a single benchmark would hide.

Step 7: Change one thing

Now the experiment becomes interesting.

Change exactly one variable:

• prompt
• model
• temperature
• retrieval configuration
• context window
• tool selection
• output schema

Then run the complete evaluation again.

Do not change five things at once.

Otherwise you won’t know what caused the result.

Step 8: Compare the versions

Create a simple comparison:

Metric	Version A	Version B
Accuracy	87%	91%
p95 latency	1.8s	2.1s
Cost / case	£0.003	£0.006
Malformed output	2	0
Critical failures	3	1

Again, these figures are illustrative.

Now you have an engineering conversation rather than a subjective debate.

Someone can argue that the additional cost is worthwhile.

Someone else can argue that the latency is too high.

But everyone is looking at the same evidence.

Step 9: Investigate the failures

This is where a useful evaluation harness becomes more than a score generator.

Export the failed cases.

Look at them individually.

Ask:

What type of failure is this?

For example:

FAIL-001
Expected: ACCOUNT_ACCESS
Actual: GENERAL_SUPPORT
Reason: ambiguous wording

FAIL-002
Expected: HIGH
Actual: MEDIUM
Reason: model underestimated urgency

FAIL-003
Expected: refusal
Actual: fabricated answer
Reason: insufficient grounding

You will often discover that several apparently different failures have the same underlying cause.

That’s valuable engineering information.

Step 10: Turn the harness into a regression test

Once the evaluation works, put it into your development workflow.

Every meaningful change should trigger it.

At minimum:

Pull request
     ↓
Build
     ↓
AI evaluation suite
     ↓
Compare with baseline
     ↓
Pass / investigate

You don’t necessarily need to block every deployment when the score moves by 0.5%.

But you should have explicit thresholds for important failures.

For example:

• overall accuracy must not fall below 90%
• critical-case accuracy must remain above 98%
• malformed structured responses must remain below 1%
• p95 latency must remain below 3 seconds

The exact thresholds belong to the application.

The important principle is that the system’s quality requirements become executable checks.

A useful extension: separate critical cases

Not all failures are equal.

Suppose your AI handles 1,000 ordinary requests and gets 10 wrong.

That may be tolerable.

But suppose one of your test cases represents a regulatory decision, a payment, an access-control change or a safety-critical instruction.

You may care much more about that one case.

Create a separate critical-case score.

For example:

Overall accuracy:       94%
Critical-case accuracy: 100%

This is one reason I prefer evaluation suites to a single benchmark number.

Averages can hide the failures you actually care about.

What you can reproduce

You don’t need a large AI platform to run this experiment.

A minimal implementation can consist of:

• an LLM API
• a JSON or CSV dataset
• a small Python or TypeScript runner
• a response validator
• a results file
• a script that calculates metrics

The architecture is roughly:

                ┌─────────────────┐
                │ Evaluation Cases│
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │ AI Application  │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │ Response        │
                │ Validation      │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │ Evaluation      │
                │ + Metrics       │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │ Results /       │
                │ Regression      │
                └─────────────────┘

The implementation details can vary.

The engineering pattern doesn’t.

Don’t let the benchmark become the product

There is a trap here too.

Once you have a number, teams can start optimising for the number.

That can be dangerous.

A model might achieve a higher benchmark score while becoming worse for real users.

Your evaluation dataset therefore needs to evolve.

Add new cases when:

• a production failure occurs
• users discover a new edge case
• policy changes
• the application gains a new capability
• a new failure mode is discovered

Your dataset should become a living representation of what the system needs to do correctly.

What I would measure in a real experiment

If I were running this inside an engineering organisation, I’d maintain a small dashboard containing:

Quality

• overall pass rate
• critical-case pass rate
• failure categories

Performance

• p50 latency
• p95 latency
• timeout rate

Economics

• cost per request
• cost per successful outcome

Reliability

• malformed responses
• retry rate
• tool failures

Change impact

• improvement over previous version
• regressions introduced
• newly discovered failures

That gives engineering, product and risk teams a common language.

The bigger lesson

The interesting thing about evaluation isn’t the benchmark itself.

It’s what happens to the development process once you have one.

Without an evaluation harness, changing an AI system often feels like this:

“I think the new prompt is better.”

With one, the conversation becomes:

“The new prompt improved classification accuracy by 4 percentage points, eliminated malformed responses, increased p95 latency by 180ms, and introduced two regressions in critical cases.”

That’s a completely different engineering conversation.

And it changes how teams work.

You can experiment faster because you can measure the consequences.

You can change models without relying entirely on vendor benchmarks.

You can catch regressions before production.

And you can explain to stakeholders why you believe a system is improving.

Try it yourself

If you want to reproduce the experiment, start small.

1. Pick one narrow AI task.
2. Collect 50–100 representative examples.
3. Define the expected result for each.
4. Run your current implementation against them.
5. Record accuracy, latency and cost.
6. Categorise failures.
7. Change exactly one thing.
8. Run the same evaluation again.
9. Compare the two versions.
10. Add the evaluation to your development workflow.

Then repeat.

The first version doesn’t need to be sophisticated.

A 100-case evaluation harness that your team actually runs is more valuable than a 10,000-case benchmark that nobody maintains.

And once you have it, you can start asking much more interesting questions about your AI systems.

Which model works best for our workload?

Which prompts are actually improving performance?

Where does retrieval help?

Where does an agent introduce unnecessary risk?

Which failures need better prompting — and which require architectural constraints?

Those are engineering questions.

And that’s where AI development starts to become engineering rather than experimentation by intuition.