Why your fine-tuning job takes longer than the FLOPs say
Why your fine-tuning job takes longer than the FLOPs say
Using 272 production supervised fine-tuning runs on NVIDIA H200 GPUs (about 7,382 GPU-hours), the authors compared analytical FLOP-based duration estimates against learned predictors. The standard FLOP estimate explained little of the observed time (R² = 0.31, MAE 12.41 h), while a CatBoost regressor using only pre-dispatch configuration features reached R² = 0.98 with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation. The authors attribute the gap to training-specific signals such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and note the model is limited to SFT on H200s with few runs above 60 hours.
Source: Hugging Face Blog · huggingface.co