pass@k
The probability that at least one of k sampled solutions is correct; pass@1 is a single attempt.
Introduced with HumanEval in 2021, pass@k estimates how often a model solves a task if allowed k tries. pass@1 is the number that matters for autonomous use; pass@k for large k measures whether the model can solve it given a verifier to choose. The difference between the two is the value of a good test suite or reviewer. Vendors' 'parallel test-time compute' scores are effectively pass@k with a learned selector.