Milestone C

Each milestone is scored out of twenty under the peer-review protocol: five criteria, 0 to 4 each. Score 0 for missing evidence, 1 to 2 for an incomplete artifact, 3 for a sound artifact with limited measurement, and 4 for reproducible evidence with the measurement, its noise and its limitations stated. 14 of 20 passes. A high score never overrides a failed mastery gate.

• Retrieval over a messy corpus with a versioned golden set and recall@k before and after one change; a permission test that proves a forbidden chunk never returns.
• An eval harness: error-analysis categories with raw examples, a code grader, a judge with TPR and TNR against 100+ labels, noise measured, a CI gate that fails on injected regressions.
• An agent with rules in code, a step cap and budget, pass@3 and pass^3 reported, and a documented list of unfixed failures with reasons.
• A cost cut: cost per request halved, the eval unchanged within the noise band, the levers named; the injection suite in CI with the trifecta broken in code; Targets 08 and 10 with every flag printing and, for each, the prediction list, invariant explanations, extra negative tests and limits paragraph in the repo.
• A fine-tune-or-not decision record with three variants and quality, cost and latency for each; at least five real users for two weeks with a changelog.

Challenge: a model upgrade raises your overall score by 5 points. Show, from your own suite, how you would know whether a safety category regressed, and what you do before rollout.

Submission checklist

For each milestone include: commit identifier; environment and setup; exact commands; the measurement, its denominator and its noise; what you expected before running; what you observed; limitations; next steps. Anything not run (a GPU run you could not afford, users you could not recruit) must say "not done", not be implied. Grade reasoning and reproducibility, not screenshots.

Use the instructor keys for module reasoning and the capstone for the final practical assessment.