That matches what I have seen. Synthetic sets get generated from roughly the same assumptions the screening rules encode, so they mostly confirm the rules against themselves. The test set with real signal is the historical false negatives, and that is exactly the set nobody will hand over.

the false negative file always exists and it always lives in a compliance folder nobody outside the team can open. every place ive worked had one and every place treated it like radioactive material

so the synthetic set is a unit test and you shipped it as a pen test

差不多就是这个意思。它能回答“改完规则有没有把正常客户挡在外面”，回答不了“坏的能不能混进来”。两个问题得用两套数据，我们之前一直拿一套在糊弄自己。