commissioned a labelled set for the first time and learned the whole thing lives or dies on the annotation guidelines. mine were four bullet points. got back exactly what four bullet points deserve 🙃
The guideline document is the actual product you are buying, not the labels. Anything a reasonable person could read two ways, forty annotators will read two ways, and you find out at merge time when the agreement score is somewhere around chance on the categories you cared about most.
four bullet points is not a spec its a vibe with numbering
The fix that worked for us was labelling two hundred items ourselves first and shipping those as worked examples rather than as rules. Prose describing an edge case is ambiguous. A decided edge case is not. It also gives you a gold set to score contributors against continuously instead of arguing about quality after the invoice arrives.
yeah im rewriting the whole thing around examples now. rule one is basically 'here are the twelve cases everyone gets wrong, this is what each one is'
