iCreatAI News
Skip to content
Trending storyUpdating

OpenAI advocates perplexity as a cross-lab shared metric

1 reports1 sources20 hr agoUpdated

Understand this story

AI overview

OpenAI has published internal perplexity data for its models since GPT-3.5 and advocates using perplexity (cross-entropy loss) as a metric for cross-model developer comparison and for setting shared safety standards. OpenAI says GPT-3.5's log-perplexity is about 0.99, GPT-4's is about 0.91, and the current loss is below 0.8; it expects new capabilities to emerge in the roughly 0.68-0.7 range. The above figures and capability predictions are all OpenAI's claims.

AI overview based on source reports · Updated 1 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 22
  1. OpenAI NewsFeatured
    Perplexity and Intelligence

    OpenAI argues that perplexity (cross-entropy loss) on held-out data should become a shared industry metric for tracking AI progress and informing safety standards, and publishes its internal pretraining perplexity figures from GPT-3.5 onward. The post ties capability milestones to log-perplexity levels—GPT-3.5 at ~0.99, GPT-4 at ~0.91, below 0.8 for recent cybersecurity and math advances—and extrapolates further transformative capabilities in the ~0.68–0.7 window. It proposes shared transparency norms and common development limits tied to perplexity until monitoring milestones such as a successor to CoT monitors are reached, while acknowledging challenges like cross-lab comparability and an unknown entropy floor.

Activity for this story

Not enough continuous observations to show a trend yet.