Logistic Regression and Machine Learning Models Cannot Discriminate Between Satisfied and Dissatisfied Total Knee Arthroplasty Patients

Published:November 01, 2021DOI:https://doi.org/10.1016/j.arth.2021.10.017

      Abstract

      Background

      Approximately 20% of total knee arthroplasty (TKA) patients are found to be dissatisfied or unsure of their satisfaction at 1-year post-surgery. This study attempted to predict 1-year post-surgery dissatisfied/unsure TKA patients with pre-surgery and surgical variables using logistic regression and machine learning methods.

      Methods

      A retrospective analysis of patients who underwent primary TKA for osteoarthritis between 2012 and 2016 at a single institution was completed. Patients were split into satisfied and dissatisfied/unsure groups. Potential predictor variables included the following: demographic information, patella re-surfaced, posterior collateral ligament sacrificed, and subscales from the Knee Society Knee Scoring System, the Knee Society Clinical Rating System, the Western Ontario and McMaster Universities Osteoarthritis Index, and the 12-Item Short Form Health Survey version 2. Logistic regression and 6 different machine learning methods were used to create prediction models. Model performance was evaluated using discrimination (AUC [area under the receiver operating characteristic curve]) and calibration (Brier score, Cox intercept, and Cox slope) metrics.

      Results

      There were 1432 eligible patients included in the analysis, 313 were considered to be dissatisfied/unsure. When evaluating discrimination, the logistic regression (AUC = 0.736) and extreme gradient boosted tree (AUC = 0.713) models performed best. When evaluating calibration, the logistic regression (Brier score = 0.141, Cox intercept = 0.241, and Cox slope = 1.31) and gradient boosted tree (Brier score = 0.149, Cox intercept = 0.054, and Cox slope = 1.158) models performed best.

      Conclusion

      The models developed in this study do not perform well enough as discriminatory tools to be used in a clinical setting. Further work needs to be done to improve the performance of pre-surgery TKA dissatisfaction prediction models.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic and Personal
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to The Journal of Arthroplasty
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Kahlenberg C.A.
        • Nwachukwu B.U.
        • Mclawhorn A.S.
        • Cross M.B.
        • Cornell C.N.
        • Padgett D.E.
        Patient satisfaction after total knee replacement: a systematic review.
        HSS J. 2018; 14: 192-201
        • Gunaratne R.
        • Pratt D.N.
        • Banda J.
        • Fick D.P.
        • Hons M.
        • Khan R.J.K.
        • et al.
        Patient dissatisfaction following total knee arthroplasty: a systematic review of the literature.
        J Arthroplasty. 2017; 32: 3854-3860
        • Canadian Institute for Health Information
        Hip and Knee Replacements in Canada: CJRR Annual Statistics Summary. CIHI, Ottawa ON2018-2019
        • Farooq H.
        • Deckard E.R.
        • Ziemba-Davis M.
        • Madsen A.
        • Meneghini R.M.
        Predictors of patient satisfaction following primary total knee arthroplasty: results from a Traditional statistical model and a machine learning algorithm.
        J Arthroplasty. 2020; 35: 3123-3130
        • Kunze K.N.
        • Akram F.
        • Fuller B.C.
        • Zabawa L.
        • Sporer S.M.
        • Levine B.R.
        Internal validation of a predictive model for satisfaction after primary total knee arthroplasty.
        J Arthroplasty. 2019; 34: 663-670
        • Baker P.N.
        • Rushton S.
        • Jameson S.S.
        • Reed M.
        • Gregg P.
        • Deehan D.J.
        Patient satisfaction with total knee replacement cannot be predicted from pre-operative variables alone: a cohort study from the National Joint Registry for England and Wales.
        Bone Joint J. 2013; 95: 1359-1365
        • Garriga C.
        • Sanchez-Santos M.T.
        • Judge A.
        • Perneger T.
        • Hannouche D.
        • Lübbeke A.
        • et al.
        Development of a model predicting non-satisfaction 1 year after primary total knee replacement in the UK and transportation to Switzerland.
        Sci Rep. 2018; 8: 1-8
        • Kunze K.N.
        • Polce E.M.
        • Sadauskas A.J.
        • Levine B.R.
        Development of machine learning algorithms to predict patient dissatisfaction after primary total knee arthroplasty.
        J Arthroplasty. 2020; 35: 3117-3122https://doi.org/10.1016/j.arth.2020.05.061
        • Ramkumar P.N.
        • Haeberle H.S.
        • Bloomfield M.R.
        • Schaffer J.L.
        • Kamath A.F.
        • Patterson B.M.
        • et al.
        Artificial Intelligence and arthroplasty at a single institution: Real-World Applications of machine learning to Big data, value-based Care, Mobile health, and Remote patient Monitoring.
        J Arthroplasty. 2019; 34: 2204-2209
        • Haeberle H.S.
        • Helm J.M.
        • Navarro S.M.
        • Karnuta J.M.
        • Schaffer J.L.
        • Callaghan J.J.
        • et al.
        Artificial Intelligence and machine learning in lower Extremity arthroplasty: a review.
        J Arthroplasty. 2019; 34: 2201-2203
        • Noble P.C.
        • Scuderi G.R.
        • Brekke A.C.
        • Sikorskii A.
        • Benjamin J.B.
        • Lonner J.H.
        • et al.
        Development of a new Knee Society scoring system.
        Clin Orthop Relat Res. 2012; 470: 20-32
        • Insall J.N.
        • Dorr D.
        • Scott R.
        • Scott W.
        Knee Society score Rationale.
        Clin Orthop Relat Res. 1989; 26: 13-14
        • Bellamy N.
        • Buchanan W.
        • Goldsmith C.
        • Campbell J.
        • Stitt L.
        Validation study of WOMAC: a health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee.
        J Rheumatol. 1988; 15: 1833-1840
        • Ware J.E.
        • Krosinski M.
        • Keller S.D.
        A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity.
        Med Care. 1996; 34: 220-233
        • Torgo L.
        Data mining with R: learning with case studies.
        1st ed. Taylor & Francis, Boca Raton, FL2010
        • Friedman J.
        • Hastie T.
        • Tibshirani R.
        Regularization paths for generalized linear models via coordinate descent.
        J Stat Softw. 2010; 33: 1-22
        • Liaw A.
        • Wiener M.
        Classification and regression by randomForest.
        R News. 2002; 2: 18-22
        • Greenwell B.
        • Boehmke B.
        • Cunningham J.
        • GBM Developers
        “Gbm: Generalized Boosted Regression Models.” R Package Version.
        2019: 2
        • Chen T.
        • He T.
        • Benesty M.
        • Khotilovich V.
        Package ‘xgboost’. R version.
        2019: 90
        • Meyer D.
        • Dimitriadou E.
        • Hornik K.
        • Weingessel A.
        • Leisch F.
        • Chang C.C.
        • Lin C.C.
        E1071: misc functions of the department of statistics, probability theory group.; 2019. R package version.
        2021: 1-6
      1. Allaire JJ, Chollet F. keras: R Interface to 'Keras'. R package version 2.2. 0.

        • James G.
        • Witten D.
        • Hastie T.
        • Tibshirani R.
        An Introduction to Statistical Learning.
        Springer Science+Business Media, New York2013
        • Kuhn M.
        • Johnson K.
        Applied Predictive Modeling.
        Springer Science+Business Media, New York2013
        • R Core Team
        R: a language and environment for statistical computing.
        R Foundation for Statistical Computing, Vienna Austria2020
        • Christodoulou E.
        • Ma J.
        • Collins G.S.
        • Steyerberg E.W.
        • Verbakel J.Y.
        • Van Calster B.
        A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.
        J Clin Epidemiol. 2019; 110: 12-22
        • Harrell F.
        Regression Modeling Strategies.
        Springer Science+Business Media, New York2015
        • Spiegelhalter D.J.
        Probabilistic prediction in patient management and clinical trials.
        Stat Med. 1986; 5: 421-433
        • Nakano N.
        • Shoman H.
        • Olavarria F.
        • Matsumoto T.
        • Kuroda R.
        • Khanduja V.
        Why are patients dissatisfied following a total knee replacement? A systematic review.
        Int Orthopaedics. 2020; 44: 1971-2007https://doi.org/10.1007/s00264-020-04607-9
        • Ramkumar P.N.
        • Navarro S.M.
        • Haeberle H.S.
        • Ng M.
        • Piuzzi N.S.
        • Spindler K.P.
        No difference in outcomes 12 and 24 Months after lower Extremity total joint arthroplasty: a systematic review and meta-analysis.
        J Arthroplasty. 2018; 33: 2322-2329