Perplexity and Intelligence
Perplexity and Intelligence
OpenAI argues that perplexity (cross-entropy loss) on held-out data should become a shared industry metric for tracking AI progress and informing safety standards, and publishes its internal pretraining perplexity figures from GPT-3.5 onward. The post ties capability milestones to log-perplexity levels—GPT-3.5 at ~0.99, GPT-4 at ~0.91, below 0.8 for recent cybersecurity and math advances—and extrapolates further transformative capabilities in the ~0.68–0.7 window. It proposes shared transparency norms and common development limits tied to perplexity until monitoring milestones such as a successor to CoT monitors are reached, while acknowledging challenges like cross-lab comparability and an unknown entropy floor.
Offers a rare look at OpenAI's internal pretraining metrics and a concrete proposal for industry-wide pacing standards built on a single measurable quantity.
Source: OpenAI News · openai.com