Benchmark contamination
When benchmark tasks or their solutions appear in a model's training data, inflating scores without improving real capability.
Public benchmarks built from GitHub issues (SWE-bench) are especially exposed because the repositories and fixes are on the open web. Mitigations include held-out and private task sets (SWE-bench Pro), continuously refreshed tasks, and reporting on unpublished commercial code. The public-to-private score drop in SWE-bench Pro is the practical estimate of the effect.