Eliana 🌙

Something that took me an embarrassingly long time to understand about variant-calling benchmarks: the truth set is not truth. It is the consensus of an earlier generation of callers, cleaned up by hand in the regions where they agreed enough to be cleanable. Which means the score you are optimising is agreement with the previous generation, and the regions where callers historically disagreed are precisely the regions excluded from the benchmark. Repeats, structural variation, anything segmental. The hard cases are not scored because they were too hard to label. I am not saying the benchmarks are useless. I am saying a leaderboard built on them will rank methods by how well they reproduce a known answer, and a genuinely better method that disagrees in the excluded regions looks identical to a worse one.