Reliability (agent)
How consistently an agent produces a correct outcome across runs and inputs, as opposed to peak capability.
Capability is what an agent can do at least once; reliability is what it does every time. Benchmarks report the former, production demands the latter. Measures include pass^k, success-rate variance across seeds, and the 80%-time-horizon (much shorter than the 50% horizon). LangChain finds quality is the top barrier to production for 32% of teams; 87% of developers are concerned about agent accuracy.