Assessing undergraduate competence in evidence based medicine: a preliminary study on the correlation between two objective instruments

CONTEXT: The Fresno test and the Berlin Questionnaire are two validated instruments for objectively assessing competence in evidence-based medicine (EBM). Although both instruments purport to assess a comprehensive range of EBM knowledge, they differ in their formats. We undertook a preliminary study using the adapted version of the two instruments to assess their correlations when administered to medical students. The adaptations were made mainly to simplify the presentation for our undergraduate students while preserving the contents that were assessed.
METHODS: We recruited final-year students from a Malaysian medical school from September 2006 to August 2007. The students received a structured EBM training program within their curriculum. They took the two instruments concurrently, midway through their final six months of training. We determined the correlations using either the Pearson's or Spearman's correlation depending on the data distribution.
RESULTS: Of the 120 students invited, 72 (60.0%) participated in the study. The adapted Fresno test and the Berlin Questionnaire had a Cronbach's alfa of 0.66 and 0.70, respectively. Inter-rater correlation (r) of the adapted Fresno test was 0.9. The students scored 45.4% on average [standard deviation (SD) 10.1] on the Fresno test and 44.7% (SD 14.9) on the Berlin Questionnaire (P = 0.7). The overall correlation between the two instruments was poor (r = 0.2, 95% confidence interval: -0.07 to 0.42, P = 0.08), and correlations remained poor between items assessing the same EBM domains (r = 0.01-0.2, P = 0.07-0.9).
DISCUSSION: The adapted versions of the Fresno test and the Berlin Questionnaire correlated poorly when administered to medical students. The two instruments may not be used interchangeably to assess undergraduate competence in EBM.