An AI system can pass a demanding test and still leave your customer confused.
That is an uncomfortable distinction for a business trying to decide whether its investment in AI is working. A score is easy to put on a dashboard. A useful outcome takes more work to define.
I have spent more than 20 years across the software development lifecycle. In my own agent-assisted development, I have seen automated tests pass while something obvious was still wrong when I used the application. That experience keeps bringing me back to a basic question: what did the test actually establish?
The same question applies beyond software quality. If an agent completes an onboarding flow, produces a proposal or recommends a next action, we need to understand the relationship between completing the task and helping the person.
The evidence has to match the claim.
Capability is only part of the answer
Andrew Ng's October 16 letter introduction draws a distinction between tasks with verifiable answers and tasks that require contextual judgment. The publicly accessible heading and introduction are enough to raise an important question for leaders: how much of a benchmark result transfers to the work we actually need done? The full letter was not available for this review. Andrew Ng, October 16, 2026.
A practitioner example makes the distinction tangible. In his October 20 account of building Battle Drones, Simon Willison describes agent-assisted game development that combined automated simulations with his own assessment of what made the game enjoyable. Agents contributed to design as well as implementation. His account is a useful example, not a controlled study of game quality. Simon Willison, October 20, 2026.
The business lesson I take from this is about choosing evidence. A working game and an enjoyable game are related, but they are different claims. An application can satisfy a set of rules without delivering the experience that motivated those rules.
I would separate evaluation into three layers.
First, checkable conditions: does the output satisfy explicit requirements? Second, contextual judgment: is it appropriate and useful in the circumstances? Third, observed outcomes: what happens when people use it?
These layers support each other. None should quietly stand in for the others.
A passing check, a useful experience and a better business outcome need different evidence.
Use this framework to examine a success claim. Identify the evidence behind it, then ask which layer remains untested. A dashboard can show several measures, but it should not turn a serious failure into an acceptable average.
Make the distinction concrete: designing onboarding
I am building a community app where people can interact and benefit each other. Consider a hypothetical onboarding flow for that kind of application. This is an illustrative design exercise, not a report of a feature I have launched.
An agent could build the screens, connect the data model, add translations and test that required fields behave correctly. Those are useful capabilities. We could also check that private information stays private and that the flow works on the intended web and mobile clients.
Now imagine every check passes, but a new member reaches the final screen and has no idea what to do next.
The implementation may meet the written requirements. The product has still left an important need unresolved.
We might tell an agent to increase completion. It could propose fewer questions and shorter explanations. That may help. It could also remove guidance that made the community understandable. A higher completion rate would then need to be interpreted alongside what members can actually do afterward.
This is where the evaluation target becomes a product decision. What do we want a new member to accomplish? Find a relevant activity? Understand how to ask for help? Decide whether participating is appropriate for them?
Those are different outcomes. They deserve different evidence.
Write an evaluation card before optimizing
For one workflow, I would write a short evaluation card with four fields.
Outcome: describe what the person should be able to accomplish. In our hypothetical example, a new member should be able to choose a useful next action without being pressured to share unnecessary information.
Checks: list conditions that can be verified directly. Required fields should work. Access rules should be respected. Translated guidance should appear on the clients we intend to support. Specify the cases being checked rather than claiming universal coverage.
Judgment: describe the tradeoffs and how they will be assessed. Is the guidance clear? Does it explain the community without overwhelming someone? Compare concrete alternatives using a rubric, with examples of what counts as helpful or confusing. Assign someone to resolve product tradeoffs.
Observation: decide what to examine in use. Where do people hesitate, ask for help or leave? What can they explain about the next step? Look at side effects as well as the desired behavior.
Write the outcome and its evidence before optimizing the workflow.
The card connects a proposed change to a decision: keep it, revise it or gather more evidence. If observation is missing, say so. A completed form supports a narrower claim than a useful first experience.
Not every workflow needs an elaborate evaluation program. The amount of evidence should reflect the consequences of being wrong. A low-stakes wording experiment and an automated decision affecting someone's access should not receive identical treatment.
Let disagreement tell you something
Judgment introduces disagreement. That is inconvenient, but it can be informative.
Suppose one reviewer prefers a shorter onboarding flow, while another wants more guidance. Asking another agent to assign a score may create a tie-breaker without explaining the difference.
I would first keep the example and both reasons. Is one reviewer missing context about the audience? Are they applying different priorities? Is the team trading a faster first session against a better understanding of how to participate?
Missing context calls for a clearer brief. Conflicting priorities call for a product decision. Neither problem is solved merely by averaging the scores.
Disagreement can reveal a missing assumption or a real product tradeoff.
After clarifying the context or priority, compare the alternatives again. If the disagreement remains material, limit automation and gather evidence from intended users. Record what was decided and what observation might change that decision.
Agreement among reviewers is useful. It is not proof that the experience works.
Agents can help evaluate judgment, too
I do not see this as a permanent boundary between what humans and agents can do.
Agents can already help develop rubrics, generate contrasting options, identify inconsistencies and challenge an evaluation. I expect more of this work to become delegable. That possibility makes it more important to understand what an evaluation means.
If an agent is assessing another agent's output, we should ask whether its judgments correspond to the outcome we care about. A fluent explanation is evidence of an explanation. We still need to examine whether the reasoning fits the case.
Human evaluation also has limitations. People bring incomplete context, inconsistent preferences and their own assumptions. A useful process makes those assumptions discussable and revisable.
The objective is to improve the quality of the decision, whichever combination of people and agents contributes to it.
What business leaders should ask for
For the next AI workflow your team proposes, ask for one example of a technically successful result that would still disappoint the user. That example will often expose what the current evaluation leaves out.
Then ask for the evaluation card, the unresolved tradeoff and the next observation that could change the team's mind.
This is a practical way to connect AI capability with business value. It gives a team permission to say that one part works while another remains uncertain. It also prevents a strong score in a convenient test from becoming a claim about an outcome nobody has examined.
A benchmark can help us choose a tool. It cannot choose the purpose of the work for us.
Define useful before you automate the pursuit of it.
