OSF releases WPS AI benchmark and leaderboard
Understand this story
Our Secure Future (OSF) has released the first public "Women, Peace and Security" (WPS) AI benchmark and leaderboard, designed to evaluate the ability of large language models to follow WPS principles in global peacekeeping scenarios. OSF says models generally exhibit a "WPS capability gap": they know the relevant concepts but do not bring them up on their own. The full benchmark includes 30 scenarios, scoring rules for 11 criteria, a taxonomy and a reliability report, and is released under a CC BY 4.0 license that explicitly permits use as training data. The benchmark is free, and anyone can use it to test AI systems or partner with OSF to conduct evaluations. The scenario set is only valid for models released before publication; OSF plans to release new scenarios regularly to keep the benchmark valid for future models.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIAI Models Know What Women, Peace and Security Is. They Just Don't Bring It Up
Our Secure Future describes building the WPS AI Leaderboard, a 30-scenario benchmark testing whether LLMs apply Women, Peace and Security principles when users don't explicitly ask, finding that frontier models' average score fell from 0.82 on cued prompts to 0.16 on sparse ones. Models also varied unpredictably in resisting adversarial requests to sideline WPS requirements, which the authors attribute to post-training rather than general capability. A multi-agent WPS workflow scored 0.84 overall with flat gender-sensitivity across tiers, though the authors stress the benchmark is not a deployment certification and cannot replace human WPS advisors.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 13 02:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.