Kaggle

Study: Cross-Validation Scores Underestimate Model Performance by Average of X% vs. Leaderboards

A new Medium article argues that cross-validation (CV) scores are often too conservative compared to public leaderboards, leading developers to underestimate their models. The author suggests that CV can be 'timid' because it doesn't account for the specific test distributions used in leaderboards, which may be easier or more aligned with real-world tasks. The piece highlights cases where models with lower CV scores outperform on leaderboards, urging practitioners to blend both metrics. This discussion is timely as AI teams increasingly rely on leaderboards like those from Hugging Face or Kaggle to benchmark models.