Evaluate an Agent

A unit spec asks whether one middleware does what it says. An eval asks whether the whole agent — system prompt, tools, and the model behind it — behaves. Brute::Eval runs real turns against a real model and grades what the turn did: which tools were called, in what order, what the answer contained, and what it cost.

A first suite

require "brute/eval"

CASES = [
  Brute::Eval::Case.new(
    "searches for what it cannot know",
    said:     "what did the Bank of England do yesterday?",
    stubs:    { "search" => RATE_DECISION },
    calls:    { "search" => { "query" => /bank|rate/i } },
    mentions: %w[4.25],
  ),
  Brute::Eval::Case.new(
    "does not search for what it already knows",
    said:   "how many minutes are there in an hour?",
    never:  %w[search],
    budget: Brute::Eval::Budget.new(iterations: 2, tool_calls: 0),
  ),
]

exit(Brute::Eval::Suite.new(agent: "agent.ru", cases: CASES).run)

Suite#run answers a process exit status — 0 when every case passed, 1 otherwise — so an eval script is a CI step.

What a case can expect

Expectations are deliberately about what the turn did, not its prose:

Argument	Passes when
said:	the prompt handed to the agent
calls:	each named tool was called (arguments matched as regex/hash subset)
never:	none of these tools were called
order:	the listed tools were called in exactly this order
mentions:	every word appears in the final reply
silent: true	the turn answered nothing (nothing to report)
absent: true	the reply signals absence (“no results”, …)
budget:	stayed inside Budget.new(iterations:, tool_calls:, tokens:, seconds:)
stubs:	canned results served instead of real tool execution
files:, conversation:	world state laid out before the turn
runs:	repeat N times; passes only if every run does — models aren’t deterministic

Anything else goes in a block, handed the transcript:

Brute::Eval::Case.new("patches the test") do |transcript|
  transcript.failures << "edited the wrong file" unless ...
end

How it observes without changing the agent

The harness subscribes to the agent’s own hook registry (agent.on(...)) — the same one any other subscriber uses. No eval-only middleware, no branch in your agent file. Tool stubs are installed on :before_tool: a call whose name has a canned result is answered without the tool ever running, so the web, calendar, or shell is replaced without building the agent differently.

The agent under test is built fresh for every attempt — from a .ru file path, or from a block that returns a new pipeline — so nothing carries between cases.

Custom worlds

Where a case wakes up is the World’s business. The default Brute::Eval::World keeps no state: said goes straight to the turn. Subclass it for deployments that deliver input some other way — an inbox on disk, a queue, a chat room:

class Room < Brute::Eval::World
  def prepare(kase)
    super.tap { |input| inbox.append(kase.said) if input.nil? }
  end
end

Brute::Eval::Suite.new(agent: "agent.ru", world: Room.new, cases: CASES).run

A world answers three things: #prepare(case) lays out state and returns what the turn starts with (or nil if delivery happened another way), #stub(agent, stubs) installs canned tool results, and #published records whatever the turn sent outward.