Hugging Face previews fine-tuning duration prediction research
Understand this story
Hugging Face has released a paper preview proposing a method to predict the end-to-end fine-tuning time of generative language models before task dispatch. The study is based on a comparison of 272 real SFT production tasks (approximately 7,382 GPU-hours, NVIDIA H200): traditional FLOP estimation could barely explain actual duration (R²=0.31, MAE 12.41 hours); a CatBoost regressor relying only on pre-dispatch information achieved R²=0.98, MAE 1.71 hours, and a median error of 0.51 hours in 5-fold cross-validation. These are results disclosed in the paper preview and await validation by the formal paper and more scenarios.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hugging Face BlogWhy your fine-tuning job takes longer than the FLOPs say
Using 272 production supervised fine-tuning runs on NVIDIA H200 GPUs (about 7,382 GPU-hours), the authors compared analytical FLOP-based duration estimates against learned predictors. The standard FLOP estimate explained little of the observed time (R² = 0.31, MAE 12.41 h), while a CatBoost regressor using only pre-dispatch configuration features reached R² = 0.98 with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation. The authors attribute the gap to training-specific signals such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and note the model is limited to SFT on H200s with few runs above 60 hours.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 19 04:00
- Change in the past 24 hours
- -51%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.