Introduction: Assessment is an integral aspect of
teaching. One-best-answer (OBA) items, if properly
constructed are able to drive learning. In-house OBA
items are notoriously poorly-constructed. The role of
a central vetting committee is to review test items and
ensure that they adhere to expected standards. Hence,
the objective of this audit is to determine whether
central vetting has improved the construct quality of
OBA items.
Methods: We audited the psychiatry end-of posting
OBA items from before and after central vetting to
compare the quality of the items before and after
central vetting was instituted. Quality was evaluated
on appropriateness of test content, items with higher
cognition and items without flaws. A standard was not
set for this first audit.
Results: Seventy six of 181 psychiatry OBAs items
retrieved from 2011 to August 2012 had undergone
first level (department) vetting only and the remainder
105 (58.0%) had two levels of vetting; department and
central vetting committee (CVC).
Appropriateness of content increased from 92.1% to
98.1%. Items with higher order thinking doubled from
21.1% to 42.9%. Items with clinical scenario increased
by 8.4% to 78.1%. Logical ordering of options however,
remained around 50%.
Two-level vetting markedly reduced problematic
lead-in questions (67.1 to 13.3%), non-homogenous
options (42.1 to 9.5%), vague and implausible options
(39.5 to 6.7%), and spelling and grammar mistakes
(19.7 to 5.7%).
Conclusion: Two-level vetting had improved the
quality of OBAs and should be continued. This could
be enhanced by training all Faculty on writing quality
OBA items and careful selection and empowerment
of CVC members. A re-audit is to be conducted after
Faculty training.