Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

Levis B 1 , Bhandari PM 1 , Neupane D 1 , Fan S 1 , Sun Y 1 , He C 1 Show all authors , Wu Y 1 , Krishnan A 1 , Negeri Z 2 , Imran M 1 , Rice DB 3 , Riehm KE 1 , Azar M 1 , Levis AW 1 , Boruff J 4 , Cuijpers P 5 , Gilbody S 6 , Ioannidis JPA 7 , Kloda LA 8 , Patten SB 9 , Ziegelstein RC 10 , Harel D 11 , Takwoingi Y 12 , Markham S 13 , Alamri SH 14 , Amtmann D 15 , Arroll B 16 , Ayalon L 17 , Baradaran HR 18 , Beraldi A 19 , Bernstein CN 20 , Bhana A 21 , Bombardier CH 15 , Buji RI 22 , Butterworth P 23 , Carter G 24 , Chagas MH 25 , Chan JCN 26 , Chan LF 27 , Chibanda D 28 , Clover K 29 , Conway A 30 , Conwell Y 31 , Daray FM 32 , de Man-van Ginkel JM 33 , Fann JR 34 , Fischer FH 35 , Field S 36 , Fisher JRW 37 , Fung DSS 38 , Gelaye B 39 , Gholizadeh L 40 , Goodyear-Smith F 16 , Green EP 41 , Greeno CG 42 , Hall BJ 43 , Hantsoo L 44 , Härter M 45 , Hides L 46 , Hobfoll SE 47 , Honikman S 36 , Hyphantis T 48 , Inagaki M 49 , Iglesias-Gonzalez M 50 , Jeon HJ 51 , Jetté N 52 , Khamseh ME 18 , Kiely KM 53 , Kohrt BA 54 , Kwan Y 55 , Lara MA 56 , Levin-Aspenson HF 57 , Liu SI 58 , Lotrakul M 59 , Loureiro SR 25 , Löwe B 60 , Luitel NP 61 , Lund C 62 , Marrie RA 63 , Marsh L 64 , Marx BP 65 , McGuire A 66 , Mohd Sidik S 67 , Munhoz TN 68 , Muramatsu K 69 , Nakku JEM 70 , Navarrete L 71 , Osório FL 25 , Pence BW 72 , Persoons P 73 , Petersen I 21 , Picardi A 74 , Pugh SL 75 , Quinn TJ 76 , Rancans E 76 , Rathod SD 77 , Reuter K 78 , Rooney AG 79 , Santos IS 68 , Schram MT 80 , Shaaban J 81 , Shinn EH 82 , Sidebottom A 83 , Simning A 31 , Spangenberg L 84 , Stafford L 85 , Sung SC 58 , Suzuki K 86 , Tan PLL 55 , Taylor-Rowan M 87 , Tran TD 37 , Turner A 88 , van der Feltz-Cornelis CM 89 , van Heyningen T 90 , Vöhringer PA 91 , Wagner LI 92 , Wang JL 93 , Watson D 94 , White J 95 , Whooley MA 96 , Winkley K 97 , Wynter K 98 , Yamada M 99 , Zeng QZ 100 , Zhang Y 101 , Thombs BD 1 , Benedetti A 102 , Depression Screening Data (DEPRESSD) PHQ Group

Affiliations 

  • 1 Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, Québec, Canada
  • 2 Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, Ontario, Canada
  • 3 Department of Psychiatry and Behavioural Neurosciences, McMaster University, Hamilton, Ontario, Canada
  • 4 Schulich Library of Physical Sciences, Life Sciences, and Engineering, McGill University, Montréal, Québec, Canada
  • 5 Department of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands
  • 6 Hull York Medical School and the Department of Health Sciences, University of York, Heslington, York, UK
  • 7 Department of Medicine, Stanford University, Stanford, California
  • 8 McGill University Libraries, Montréal, Québec, Canada
  • 9 Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada
  • 10 Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland
  • 11 Department of Applied Statistics, Social Science, and Humanities, New York University, New York
  • 12 Department of Applied Health Sciences, School of Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK
  • 13 Department of Biostatistics and Health Informatics, King's College London, London, UK
  • 14 Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
  • 15 Department of Rehabilitation Medicine, University of Washington, Seattle
  • 16 Department of General Practice and Primary Health Care, University of Auckland, Auckland, New Zealand
  • 17 Louis and Gabi Weisfeld School of Social Work, Bar Ilan University, Ramat Gan, Israel
  • 18 Endocrine Research Center, Institute of Endocrinology and Metabolism, Iran University of Medical Sciences, Tehran, Iran
  • 19 Kbo-Lech-Mangfall-Klinik Garmisch-Partenkirchen, Klinik für Psychiatrie, Psychotherapie and Psychosomatik, Lehrkrankenhaus der Technischen Universität München, Munich, Germany
  • 20 University of Manitoba IBD Clinical and Research Centre, Winnipeg, Manitoba, Canada
  • 21 Centre for Rural Health, School of Nursing and Public Health, College of Health Sciences, University of KwaZulu-Natal, Durban, KwaZulu-Natal, South Africa
  • 22 Department of Psychiatry, Hospital Mesra Bukit Padang, Sabah, Malaysia
  • 23 Centre for Epidemiology and Population Health, The Australian National University, Canberra, Australia
  • 24 Centre for Brain and Mental Health Research, University of Newcastle, Newcastle, New South Wales, Australia
  • 25 Department of Neurosciences and Behavior, Ribeirão Preto Medical School, University of São Paulo, Ribeirão Preto, São Paulo, Brazil
  • 26 Department of Medicine and Therapeutics, Hong Kong Institute of Diabetes and Obesity and Li Ka Shing Institute of Health Science, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, Hong Kong Special Administrative Region, China
  • 27 Department of Psychiatry, National University of Malaysia, Kuala Lumpur, Malaysia
  • 28 Department of Community Medicine, University of Zimbabwe, Harare, Zimbabwe
  • 29 School of Medicine and Public Health, University of Newcastle, Callaghan, New South Wales, Australia
  • 30 School of Nursing, Queensland University of Technology, Brisbane, Queensland, Australia
  • 31 Department of Psychiatry, University of Rochester Medical Center, Rochester, New York
  • 32 Institute of Pharmacology, School of Medicine, University of Buenos Aires, Buenos Aires, Argentina
  • 33 Leids University Medical Center, Leiden, the Netherlands
  • 34 Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle
  • 35 Center for Patient-Centered Outcomes Research, Department of Psychosomatic Medicine, Center for Internal Medicine and Dermatology, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany
  • 36 Perinatal Mental Health Project, Alan J Flisher Centre for Public Mental Health, Department of Psychiatry and Mental Health, University of Cape Town, Cape Town, South Africa
  • 37 Global and Women's Health, Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
  • 38 Department of Developmental Psychiatry, Institute of Mental Health, Singapore
  • 39 Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
  • 40 Faculty of Health, University of Technology Sydney, Sydney, New South Wales, Australia
  • 41 Duke Global Health Institute, Duke University, Durham, North Carolina
  • 42 School of Social Work, University of Pittsburgh, Pittsburgh, Pennsylvania
  • 43 Center for Global Health Equity, New York University Shanghai, Shanghai, China
  • 44 Department of Psychiatry and Behavioral Sciences, The Johns Hopkins University School of Medicine, Baltimore, Maryland
  • 45 Department of Medical Psychology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
  • 46 School of Psychology, The University of Queensland, Brisbane, Queensland, Australia
  • 47 STAR-Stress, Anxiety and Resilience Consultants, Chicago, Illinois
  • 48 Department of Psychiatry, Faculty of Medicine, School of Health Sciences, University of Ioannina, Ioannina, Greece
  • 49 Department of Psychiatry, Faculty of Medicine, Shimane University, Izumo, Shimane, Japan
  • 50 Department of Psychiatry, Hospital Universitari Germans Trias i Pujol, Badalona, Spain
  • 51 Department of Psychiatry, Depression Center, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea
  • 52 Department of Clinical Neurosciences, University of Calgary, Calgary, Alberta, Canada
  • 53 School of Health and Society and School of Mathematics and Applied Statistics, University of Wollongong, Wollongong, New South Wales, Australia
  • 54 Center for Global Mental Health Equity, The George Washington University, Washington, DC
  • 55 Department of Psychological Medicine, Tan Tock Seng Hospital, Singapore
  • 56 Instituto Nacional de Psiquiatría Ramón de la Fuente Muñiz, San Lorenzo Huipulco, Tlalpan, Mexico
  • 57 Department of Psychology, University of North Texas, Denton
  • 58 Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore
  • 59 Department of Psychiatry, Faculty of Medicine, Ramathibodi Hospital, Mahidol University, Bangkok, Thailand
  • 60 Department of Psychosomatic Medicine and Psychotherapy, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
  • 61 Research Department, TPO Nepal, Kathmandu, Nepal
  • 62 Centre for Global Mental Health, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
  • 63 Departments of Medicine and Community Health Sciences, Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba, Canada
  • 64 Baylor College of Medicine, Houston and Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas
  • 65 National Center for PTSD at Veterans Affairs Boston Healthcare System, Boston, Massachusetts
  • 66 College of Nursing, University of South Florida, Tampa
  • 67 Department of Psychiatry, Faculty of Medicine and Health Sciences, Universiti Putra Malaysia, Serdang, Selangor, Malaysia
  • 68 Post-Graduate Program in Epidemiology, Federal University of Pelotas, Pelotas, Rio Grande do Sul, Brazil
  • 69 Niigata Seiryo University Health Service Center, Niigata, Japan
  • 70 Butabika National Referral Teaching Hospital, Kampala, Uganda
  • 71 Department of Epidemiology and Psychosocial Research, Instituto Nacional de Psiquiatría Ramón de la Fuente Muñiz, Ciudad de México, México
  • 72 Department of Epidemiology, Gillings School of Global Public Health, The University of North Carolina at Chapel Hill, Chapel Hill
  • 73 Department of Psycho-Pedagogic Psychiatry, Healthcare Group Sint-Kamillus, Broeders van Liefde, Bierbeek, Belgium
  • 74 Centre for Behavioural Sciences and Mental Health, Italian National Institute of Health, Rome, Italy
  • 75 Department of Statistics, American College of Radiology, NRG Oncology Statistics and Data Management Center, Philadelphia, Pennsylvania
  • 76 Institute of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, Scotland, UK
  • 77 Department of Population Health, London School of Hygiene and Tropical Medicine, London, UK
  • 78 Group Practice for Psychotherapy and Psycho-oncology, Freiburg, Germany
  • 79 Division of Psychiatry, Royal Edinburgh Hospital, The University of Edinburgh, Edinburgh, Scotland, UK
  • 80 Department of Internal Medicine, Maastricht University Medical Center, Maastricht, the Netherlands
  • 81 Department of Family Medicine, School of Medical Sciences, Universiti Sains Malaysia, Kelantan, Malaysia
  • 82 Department of Behavioral Science, The University of Texas M.D. Anderson Cancer Center, Houston
  • 83 Allina Health, Minneapolis, Minnesota
  • 84 Department of Medical Psychology and Medical Sociology, University of Leipzig, Leipzig, Germany
  • 85 Melbourne School of Psychological Sciences, The University of Melbourne, Melbourne, Victoria, Australia
  • 86 Department of General Medicine, Asahikawa University Hospital, Asahikawa, Hokkaido, Japan
  • 87 Institute of Health and Wellbeing, University of Glasgow, Glasgow, Scotland, UK
  • 88 IMPACT-the Institute for Mental and Physical Health and Clinical Translation, School of Medicine, Deakin University, Geelong, Victoria, Australia
  • 89 Department of Health Sciences, Hull York Medical School, University of York, York, UK
  • 90 Justice and Violence Prevention Programme, Institute for Security Studies, Pretoria, South Africa
  • 91 Department of Psychiatry and Mental Health, Clinical Hospital, Universidad de Chile, Santiago, Chile
  • 92 Department of Health Policy and Management, Gillings School of Global Public Health, The University of North Carolina at Chapel Hill, Chapel Hill
  • 93 Department of Community Health and Epidemiology, Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada
  • 94 Department of Psychology, University of Notre Dame, Notre Dame, Indiana
  • 95 School of Medicine and Public Health, College of Health, Medicine and Wellbeing, University of Newcastle, New South Wales, Australia
  • 96 Department of Medicine, University of California San Francisco, San Francisco
  • 97 Florence Nightingale Faculty of Nursing, Midwifery and Palliative Care, King's College London, London, UK
  • 98 School of Clinical Sciences, Monash University, Melbourne, Victoria, Australia
  • 99 Department of Pathophysiology, Tokyo Kasei Gakuin University, Chiyoda-ku, Tokyo, Japan
  • 100 Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China
  • 101 Department of Medicine and Therapeutics, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, China
  • 102 Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montréal, Québec, Canada
JAMA Netw Open, 2024 Nov 04;7(11):e2429630.
PMID: 39576645 DOI: 10.1001/jamanetworkopen.2024.29630

Abstract

IMPORTANCE: Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates.

OBJECTIVE: To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates.

DESIGN, SETTING, AND PARTICIPANTS: This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled.

MAIN OUTCOMES AND MEASURES: For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population.

RESULTS: The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes.

CONCLUSIONS AND RELEVANCE: This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.

* Title and MeSH Headings from MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine.

MeSH terms
Similar publications