Evals Before Agents
Write small pass/fail cases before adding tools, memory, autonomy, or impressive-looking agent loops.
An agent gets safer when its first promise fits in a row.
The Rule Of Thumb
Before adding tools, memory, or autonomy, write a few examples of what the system must do and must not do. Include a missing source, an unsafe request, stale information, and a confused user.
For each case, record the task, expected output, forbidden action, source requirement, rollback, and review path. A confident answer is not a pass condition.
Why This Is Not Anti-Agent
Agents are useful when a task needs state, tools, planning, or interaction with a changing environment. Evals make that behavior inspectable.
A Tiny Eval Case
Ask for a scholarship for a final-year student. A passing answer identifies the official page, eligibility, deadline, and uncertainty. If it cannot establish a current deadline, it says so. It does not invent an amount.
The Row I Want To See
Give the case a name and keep the input, expected behavior, observed failure, repair, and rerun state together. If the answer invented an amount, require source-backed amounts and rerun the same case. Record pass or blocked, not just that the prompt was changed.
When The Agent Is Worth Adding
After the first cases pass, add a useful action: checking changed pages, drafting a checklist, comparing sources, or preparing a follow-up.
Keep the receipts and rerun the cases as the system changes. A small script with five useful evals may be a better starting point than an agent with none.