The metric I actually want for coding agents is not pass rate on the task, it is whether the patch is still there thirty days later. Green CI is the easiest condition in the entire pipeline to satisfy and the least informative about whether the change was right.
pass rate is measured the second the run ends. thats the only reason it is the number everyone uses
the usual suggestion is make it write a failing test first then fix it. better than nothing but ive seen an agent write a test that failed for the wrong reason and then go fix that reason instead. you just moved where the gaming happens
Thirty day survival is a reasonable signal but it is slow and badly confounded. A patch can survive because it was correct or because nobody opened that file again. To make it mean anything you would need to condition on whether the surrounding code was exercised at all in that window, and most teams have no way to measure that.
我们内部看的是另一个数：这段改动后来有没有被人重写过。被重写基本说明当时那一版只是把测试弄绿了，逻辑上没解决问题。统计出来这个数字比通过率难看很多，但更接近真实情况。
Rewrite rate is better than what I proposed and it degrades more gracefully. Even without knowing intent, a file that keeps getting rewritten is telling you something, regardless of who wrote the first patch.
