SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies

INTRODUCTION: Large language models (LLMs) can render structured questionnaire variables as natural-language profiles, but their value in mental-health screening cannot be judged from a single fixed-threshold score. We evaluated LLM prompt conditions in a cross-questionnaire benchmark for predicting screen-positive status on the Self-Rating Depression Scale (SDS) among undergraduates. The intended application was an offline methodological benchmark using previously collected questionnaire data, not a shorter alternative to direct SDS administration.

METHODS: De-identified records were assigned to Development (n=1,951), Test (n=2,186), and Holdout (n=2,258) grade cohorts. We evaluated logistic regression, XGBoost, and 12 model-prompt conditions at the fixed 0.5 threshold and at thresholds selected only in Development-validation. We also examined paired bootstrap differences, ranking stability, scale correlations, grouped ablations, cohort shift, calibration intercepts and slopes, alternative expected calibration error specifications, representative repeated calls at temperatures 0 and 0.1, sex-stratified performance, and per-1,000 screening scenarios.

RESULTS: At the fixed 0.5 threshold on Test, GPT E1, Gemini E2, and XGBoost achieved macro-F1 values of 0.732, 0.730, and 0.674. At Development-derived thresholds of 0.38, 0.50, and 0.174, the corresponding macro-F1 values were 0.702, 0.730, and 0.660. At the Development-selected specificity target, XGBoost achieved a Test macro-F1 of 0.738. Pairwise intervals showed no clear fixed-0.5 macro-F1 difference among the three highest-ranked LLM conditions. XGBoost yielded the lowest Brier scores. The combined-sample Spearman correlation between SDS and SAS was 0.60. Together with the ablation results, this correlation indicated predictive information distributed across overlapping questionnaire groups rather than dependence on a single measured group. In repeated calls across the complete Holdout cohort, inconsistent-case rates were 23.6% for GPT E1 and 1.5% for Gemini E2. Temperature 0 did not clearly eliminate GPT E1 instability in the representative 400-case subset.

DISCUSSION: Model conclusions depended on both the threshold and the metric. Certain LLM prompt conditions were competitive at specific operating points, but the evidence did not establish a stable or general advantage over structured-data models. Nor did it establish external validity, clinical diagnostic utility, or reduced questionnaire burden. Threshold choice, calibration, output validity, and repeated-call stability therefore require separate evaluation before prospective use.