Leeroy: Making AI agents play World of Warcraft

A software engineer describes LEEROY, a benchmark in which LLM-based agents play World of Warcraft on an AzerothCore private server by writing Lua scripts that control a headless game client in real time, since agents act in turns of 2 to 60 seconds. The SDK exposes only what a human player can see and do, so coordination such as grouping and crowd-control calls must be implemented by the agents themselves, and the author settled on XP per hour as the best scoring metric after six iterations of the judge program. A group of five agents that were not told to cooperate formed a party 29 seconds into their first run and cleared the Deadmines in 17 hours 41 minutes; the source code is on GitHub and results are at leeroy.dev.