Abobaker Ahmed Khidir Hassan

Artificial Intelligence Specialization · Cohort 2026

Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling

Multiple-choice benchmarks are the backbone of language model evaluation because they are cheap and exactly gradable, yet frontier models now crowd 90% on them, and the standard remedy, authoring harder items, is expensive, slow, and restarts the contamination clock. We propose answer pooling, a transform that makes existing annotations harder instead of writing new ones. Pool the options of N questions that share a document into one exclusive candidate list and ask the model for the complete assignment. Deleting gold answers from the pool further manufactures unanswerable questions at any ratio, so abstention is scored jointly with accuracy in the same forward pass, with no judge and no new annotation. On QuALITY, the best evaluated model falls from 0.91 (MCQ) to 0.52 (exact assignment), the guessing floor drops from 0.25 to ≈ 10−7, and model separation doubles, while rankings are preserved and a choices-only baseline is provably at chance. Withholding answers then exposes what accuracy cannot see: two models with statistically tied accuracy differ by 0.42 in false-answer rate, and our weakest model declined 2 of 900 unanswerable questions. Most surprising, the measured rate belongs to the instrument as much as to the model. Printing a none option halves false answers, multiplies false abstentions by 2 to 4, and flips a significant accuracy ranking between two models. Any benchmark with several questions per context can be revived and audited this way for free, and abstention results are comparable only under a stated response format.

Read the final report (PDF)