Classifying coronary heart disease risk from NHANES survey data (2011-2018), with a full leakage audit and calibration check [P]
Our take
The recent project by /u/YouJonaa, detailed in their Github repository, offers a valuable and refreshingly transparent look at the challenges of predictive modeling in healthcare, specifically concerning coronary heart disease (CHD) risk. Their work, utilizing the National Health and Nutrition Examination Survey (NHANES) data, isn't just about achieving high accuracy scores; it’s a compelling case study in responsible AI development and data handling. It builds upon existing work exploring similar datasets, like this analysis of NHANES data to predict diabetes Predicting Diabetes, and complements efforts focused on cardiovascular risk assessment using machine learning Cardiovascular Risk. What distinguishes this project is the explicit and thorough examination of data leakage and probability calibration, areas often glossed over in the pursuit of impressive metrics. The focus on these often-overlooked aspects makes this a particularly insightful contribution to the field.
The most significant takeaway from this project isn’t necessarily the final ROC-AUC of 0.875, but rather the meticulous investigation of spurious correlations. The observation that including other cardiovascular diagnoses significantly inflated the PR-AUC, only to reveal the model was essentially learning that "people who report one cardiovascular issue tend to report others," is a critical lesson for anyone building predictive models in healthcare. This highlights the inherent risks of relying solely on correlation without understanding the underlying causal relationships. The author’s decision to document this leakage effect, rather than simply removing the problematic variables, demonstrates a commitment to transparency and scientific rigor. Furthermore, the recalibration step, addressing the miscalibration of raw probabilities inherent in the class-weighted logistic regression, showcases a deep understanding of the importance of reliable probability estimates for clinical decision-making. This is especially relevant given the low prevalence of CHD in the dataset, where accurate probability assessment is vital.
The author’s honest assessment of the model’s limitations – specifically, the low Positive Predictive Value (PPV) – further reinforces the project's value. Acknowledging that most positive predictions are incorrect, given the rarity of CHD, is far more useful than presenting inflated metrics that could mislead users. The author’s roadmap for future improvements, including the incorporation of smoking status, diabetes, and blood pressure medication use, suggests a continued commitment to refining the model and addressing remaining gaps in predictive power. The choice to freeze the decision threshold on the development set before evaluating the test set is also a best practice, preventing the common pitfall of inadvertently tuning the model to the test data and inflating performance estimates. This methodical approach to model building, emphasizing validation and avoiding overfitting, is a hallmark of responsible AI development.
Looking ahead, it’s worth considering how these lessons in data leakage and calibration can be applied more broadly across healthcare AI. The increasing availability of large datasets, like NHANES, presents both opportunities and challenges. While these datasets offer the potential to build powerful predictive models, they also contain inherent biases and complexities that require careful consideration. As AI continues to permeate healthcare, the emphasis should shift from simply achieving high accuracy to building models that are transparent, reliable, and clinically useful – a goal that this project admirably embodies. The question now is: how can we incentivize and standardize these rigorous data validation and calibration practices across the broader AI healthcare landscape, ensuring that models are not only accurate but also trustworthy and ethically sound?
Github repo :
https://github.com/YouCele/nhanes-chd-classification
I built a project that predicts self-reported, physician-diagnosed coronary heart disease using four cycles of NHANES data (2011-2012 to 2017-2018), about 21,500 adults after cleaning. It compares logistic regression with random forest and gradient boosting, trained on demographics, blood pressure, body measurements and a lipid panel.
NHANES has a questionnaire section that asks directly about other cardiovascular diagnoses, like stroke, heart attack and angina, along with the CHD outcome itself. If I included those variables, PR-AUC jumped from 0.23 to 0.51. That's mostly the model learning that people who report one cardiovascular diagnosis tend to report others too, not something about real risk factors. So I removed that whole section and wrote up how big the leakage effect was, instead of just quietly dropping the columns.
The class weighted logistic regression, needed because CHD prevalence is only about 4%, gave badly miscalibrated raw probabilities. The mean predicted risk was close to 30% while the real rate was 4%. I fixed this with a sigmoid recalibration fit on the development set, before the test set was touched.
I also picked and froze the decision threshold on the dev set before looking at the test set, so the reported test metrics aren't tuned after the fact.
Final results on the held out test set: ROC-AUC 0.875 and PR-AUC 0.239 for logistic regression, and random forest and gradient boosting came out about the same. Age alone gets 0.83 AUC. Blood pressure, cholesterol and body size explain most of what's left. PPV at the chosen threshold is 0.13, so most positive predictions are wrong, which is expected given how rare CHD is in this data, and I state this directly in the report instead of leaving it out.
Smoking status, diabetes and blood pressure medication use aren't in the current features. NHANES has all three, I just haven't added them yet.
Happy to hear feedback, especially on the calibration step or anything in how I handled the leakage that looks wrong
[link] [comments]
Read on the original site
Open the publisher's page for the full experience