•1 min read•from Data Science
Best technique for training models on a sample of data?
Our take
Training machine learning models on a sample of data, especially in the context of imbalanced datasets, requires careful consideration. Under-sampling the majority class can help manage memory limitations, but it’s essential to implement robust cross-validation techniques to ensure model reliability. Utilize a holdout dataset for final evaluation, testing your model on unsampled data to assess its generalization capabilities. This approach not only helps in selecting the best-performing model but also ensures that your findings are applicable to the broader dataset, enhancing overall insights and productivity.
Due to memory limits on my work computer I'm unable to train machine learning models on our entire analysis dataset. Given my data is highly imbalanced I'm under-sampling from the majority class of the binary outcome.
What is the proper method to train ML models on sampled data with cross-validation and holdout data?
After training on my under-sampled data should I do a final test on a portion of "unsampled data" to choose the best ML model?
[link] [comments]
Read on the original site
Open the publisher's page for the full experience
Related Articles
- Rescaling logistic regression predictions for under-sampled data?I'm building a predictive model for a large dataset with a binary 0/1 outcome that is heavily imbalanced. I'm under-sampling records from the majority outcome class (the 0s) in order to fit the data into my computer's memory prior to fitting a logistic regression model. Because of the under-sampling, do I need to rescale the model's probability predictions when choosing the optimal threshold or is the scale arbitrary? submitted by /u/RobertWF_47 [link] [comments]
- Retraining strategy with evolving classes + imbalanced labels?Hi all — I’m looking for advice on the best retraining strategy for a multi-class classifier in a setting where the label space can evolve. Right now I have about 6 labels, but I don’t know how many will show up over time, and some labels appear inconsistently or disappear for long stretches. My initial labeled dataset is ~6,000 rows and it’s extremely imbalanced: one class dominates and the smallest class has only a single example. New data keeps coming in, and my boss wants us to retrain using the model’s inferences plus the human corrections made afterward by someone with domain knowledge. I have concerns about retraining on inferences, but that's a different story. Given this setup, should retraining typically use all accumulated labeled data, a sliding window of recent data, or something like a recent window plus a replay buffer for rare but important classes? Would incremental/online learning (e.g., partial_fit style updates or stream-learning libraries) help here, or is periodic full retraining generally safer with this kind of label churn and imbalance? I’d really appreciate any recommendations on a robust policy that won’t collapse into the dominant class, plus how you’d evaluate it (e.g., fixed “golden” test set vs rolling test, per-class metrics) when new labels can appear. submitted by /u/fleeced-artichoke [link] [comments]
- Precision and recall > .90 on holdout dataI'm running ML models (XGBoost and elastic net logistic regression) predicting a 0/1 outcome in a post period based on pre period observations in a large unbalanced dataset. I've undersampled from the majority category class to achieve a balanced dataset that fits into memory and doesn't take hours to run. I understand sampling can distort precision or recall metrics. However I'm testing model performance on a raw holdout dataset (no sampling or rebalancing). Are my crazy high precision and recall numbers valid? Of course there could be something fishy with my data, such as an outcome variable measuring post period information sneaking into my variable list. I think I've ruled that out. submitted by /u/RobertWF_47 [link] [comments]