An LLM game benchmark answers a question static tests struggle with: how does a model reason when every decision has consequences and the situation keeps changing? Multiple-choice benchmarks can leak into training data and slowly turn into memory tests. A game of chess, poker or Dou Dizhu is different every time it is played. This article explains why games are such a revealing way to compare AI models, and the specific rules AI Game Arena uses to keep the comparison fair for every model.
Why use games to benchmark LLMs?
Games have four properties that make them unusually good tests of general reasoning:
- They are fresh. A shuffled deck or a new opening produces positions the model has never seen, so it cannot rely on a memorized answer.
- Every decision is checkable. The rules are exact. A move is legal or it is not, and the result of the game is not a matter of opinion.
- They reward planning. Winning takes more than one good answer; a model has to keep a strategy together over dozens of decisions.
- They cover different kinds of thinking. Chess and Xiangqi test calculation with perfect information. Texas Hold'em and Blackjack test probability and risk with hidden cards. Dou Dizhu adds teamwork, because two players must cooperate against a third.
Games are also easy to understand for people who are not machine-learning researchers. Anyone can watch a model blunder its queen or fold a winning hand, and immediately understand what went wrong.
What makes an LLM game benchmark unfair
It is surprisingly easy to build a game benchmark that measures the wrong thing. The most common problems are:
- Model-specific prompts. If one model gets a hand-tuned prompt and another gets a generic one, you are measuring prompt engineering, not the model.
- Hidden hints. Telling a model that "this position is dangerous" or "this move looks strong" means the harness is doing part of the thinking.
- Random fallback moves. Many harnesses play a random legal move when a model fails. A model can then win games with moves it never chose.
- Incomplete state. Forcing a model to reconstruct the board from memory mostly tests bookkeeping, and punishes models for errors that have nothing to do with strategy.
The rules we use to keep the arena fair
AI Game Arena is built around one principle: the referee supplies data and enforces rules, but never thinks for anyone.
1. One prompt for every model
Every chat model receives exactly the same system prompt, the same rules text and the same data format. Nothing is tuned for a particular vendor. The only thing that changes between seats is the player's name. Changing the interface language on the site does not change what the models see either; the prompt stays byte-for-byte identical.
2. Complete facts, zero opinions
Each model gets the full game state from its own point of view: its hand or pieces, the complete move history and whose turn it is. It also gets objective facts the referee can compute directly from the rules, such as the list of legal moves and the immediate result of each one ("this Othello move flips 4 discs", "this play leaves you with 6 cards"). It never gets an evaluation, a score or a recommended move. Hidden information stays hidden: a poker model cannot see its opponents' cards.
3. No random moves, ever
If a model returns an invalid move, or its API times out or errors, the match simply pauses on that turn. The error is shown in the match log with a Retry button, and the retry sends exactly the same request as the first attempt. The model is not told why its previous answer failed, and the system never plays a move on its behalf. Every move in a finished game is one the model actually chose.
What an LLM game benchmark measures
Alongside the result of each match, the arena records statistics for every seat in real time:
| Metric | What it tells you |
|---|---|
| Wins and final score | Overall playing strength in that game |
| Decisions | How many moves the model actually had to think about |
| Average response time | Latency: fast models versus slow, heavy reasoning models |
| Invalid moves | How reliably the model follows the rules and output format |
| API failures | Timeouts and errors from the model provider |
Each decision also keeps the model's written reasoning and a short public comment, so you can read why it played what it played. A finished match can be exported as a PDF report with screenshots of every move, or recorded as a video.
Limits of game-based benchmarks
Games are revealing, but they are not the whole story. A single match involves a lot of luck, especially in card games, so conclusions need many games rather than one. Skill in chess does not guarantee skill at writing code or summarizing documents. And results depend on the model version and settings on the day you test. Treat the arena as a transparent, repeatable way to watch models reason, not as a final ranking of which AI is "best".
Run your own LLM game benchmark
You can run these comparisons yourself in the browser. Pick a game from the lobby, seat any combination of AI players, and watch. Good starting points:
- Chess for calculation and rule-following. See Can LLMs play chess?
- Texas Hold'em for probability, bluffing and chip management.
- Dou Dizhu for hidden information and teamwork. New to it? Start with our Dou Dizhu rules guide.
- Gomoku for fast pattern recognition.
You can also add your own model, using an OpenAI, Anthropic, Gemini or any OpenAI-compatible API key, and seat it next to the built-in players under exactly the same rules.