Project: Calibration Harness

Every threshold in this course so far has been a number somebody picked. 0.65 for a category, 0.85 for a transfer, 0.4 for a self-harm signal. Those are reasonable starting values and they are all, strictly, vibes. This project replaces them with evidence from your own data.

The question is calibration: when Jev reports 0.9 confidence, is it right about 90% of the time? A well-calibrated model produces a table where accuracy tracks confidence bucket for bucket. A poorly calibrated one produces a flat line, and a flat line means your gate is doing nothing but adding latency.

You need labels, and there is no way around it

A hundred examples is enough to see the shape. Two hundred gives you buckets that are not embarrassingly thin at the top. Draw them from production traffic rather than writing them, because the whole point is to measure the distribution you will actually see, and label them by hand before you look at what Jev said.

# labelled.jsonl — one object per line
{"id": "t-0001", "state": {"ticket": {"text": "..."}}, "truth": "billing"}
{"id": "t-0002", "state": {"ticket": {"text": "..."}}, "truth": "bug"}