Game Theory and AI: Nash Equilibrium, Poker and Self-Play
Quick answer: AI uses game theory to choose strategies when outcomes depend on other players’ decisions. It models possible actions, calculates expected payoffs and sometimes randomizes its choices so it cannot be predicted. A key concept is the Nash equilibrium, where no player can improve their expected payoff by changing strategy alone. Poker AIs such as Libratus (2017) and Pluribus (2019) used self-play and counterfactual regret minimization to build strong strategies despite hidden information and beat top professionals. Game theory also guides AI in negotiation, security and competition.
Facts checked November 22, 2026.
Chess and Go have perfect information: both players see the whole board. Poker, negotiations, auctions and cybersecurity do not. There, the best move depends on what the other side might do, and the other side is thinking the same about you. The mathematics of these situations is game theory, and it is behind some of AI’s most striking wins.
The Basic Vocabulary
| Term | Meaning |
|---|---|
| Players | The decision makers |
| Strategies | The choices available to each player |
| Payoffs | The numerical outcome for each combination of choices |
| Pure strategy | Always making the same choice |
| Mixed strategy | Choosing randomly with set probabilities |
| Nash equilibrium | Strategies where no player can do better by changing alone |
John Nash proved in 1950 that every finite game has at least one equilibrium, possibly in mixed strategies. Earlier, in 1928, John von Neumann proved the minimax theorem for two-player zero-sum games: there is a single value each side can guarantee. For a classic two-player example, see the prisoner’s dilemma explained.
Why Randomness Can Be the Best Strategy
In rock-paper-scissors, any predictable pattern can be exploited. The equilibrium is to play each option with probability \(\tfrac{1}{3}\), which gives an expected payoff of 0 against any opponent. Being unpredictable is not a weakness; it is the optimal strategy. Poker bluffing works the same way: a player who never bluffs is easy to read.
Worked Example: The Penalty Kick
A kicker chooses left (L) or right (R); the goalkeeper dives left or right. The table shows the chance of a goal:
| Goalie dives L | Goalie dives R | |
|---|---|---|
| Kick L | 0.5 | 0.9 |
| Kick R | 0.8 | 0.4 |
Let \(p\) be the probability the kicker kicks left and \(q\) the probability the goalie dives left. Neither pure choice is safe, so both mix. At equilibrium, each player randomizes so the other is indifferent. If the goalie dives left with probability \(q\), the kicker’s scoring chances are equal when
\[ 0.5q + 0.9(1 – q) = 0.8q + 0.4(1 – q) \;\Rightarrow\; 0.9 – 0.4q = 0.4 + 0.4q \;\Rightarrow\; q = 0.625. \]If the kicker kicks left with probability \(p\), the goalie is indifferent when
\[ 0.5p + 0.8(1 – p) = 0.9p + 0.4(1 – p) \;\Rightarrow\; 0.8 – 0.3p = 0.4 + 0.5p \;\Rightarrow\; p = 0.5. \]The goal probability at equilibrium is \(0.5 \times 0.5 + 0.8 \times 0.5 = 0.65\), whichever way the goalie dives. Neither side can raise its result by changing alone. These numbers are hypothetical, but a 2002 study by Pierre-André Chiappori, Steven Levitt and Timothy Groseclose found real professional penalty kicks consistent with several mixed-strategy predictions. Brush up on probability if the expected values are new.
How AI Bluffs: The Math of a Balanced Bet
On the last betting round of a poker hand, suppose the pot holds 100 chips and a player bets 100. The opponent must call 100 to win 200 (the pot plus the bet). If a fraction \(b\) of the player’s bets are bluffs, calling gains 200 against a bluff and loses 100 against a strong hand. The caller is indifferent when
\[ 200b = 100(1 – b) \quad \Rightarrow \quad b = \tfrac{1}{3}. \]So a balanced strategy bluffs about one bet in three in this spot. Bluff more and the opponent should always call; bluff less and they should always fold. Poker AIs compute balances like this across millions of situations. A strategy’s exploitability measures how much a perfect opponent could win against it; strong poker AIs aim to keep it near zero.
How AI Finds Equilibria: Self-Play
Real games are far too large to solve by hand: heads-up no-limit Texas hold’em has an astronomically large number of possible betting situations, so even computers must use simplified versions of the game. AI systems find good strategies by playing against copies of themselves millions of times and adjusting. A key algorithm, counterfactual regret minimization (CFR), tracks for each decision how much better each alternative would have done (“regret”) and shifts probability toward the choices it regrets not making. In two-player zero-sum games, the average strategy provably approaches a Nash equilibrium; in games with more players, self-play gives no such guarantee, though it often works well in practice. This is a cousin of reinforcement learning.
Milestones: Game Theory Meets AI
| Year | System | Achievement |
|---|---|---|
| 2015 | Cepheus (University of Alberta) | Essentially (weakly) solved heads-up limit Texas hold’em |
| January 2017 | Libratus (Carnegie Mellon) | Beat four top professionals over 120,000 hands of heads-up no-limit hold’em |
| 2019 | Pluribus (Carnegie Mellon and Facebook AI) | Beat elite professionals in six-player no-limit hold’em, where Nash guarantees no longer apply cleanly |
| 2022 | Cicero (Meta) | Ranked in the top 10% of online Diplomacy participants who played more than one game, negotiating in natural language |
Pluribus is notable because six-player poker is not zero-sum between two sides, so the clean equilibrium guarantees break down; self-play strategies still proved strong in practice. Cicero combined a language model with strategic planning to negotiate, make deals and keep promises with human players.
Game Theory vs Reinforcement Learning
Reinforcement learning is about one agent learning from rewards; game theory is about several decision makers whose rewards depend on each other. They meet in self-play: each copy of the AI is learning by reinforcement while the others form its changing environment, and game theory tells us what a stable outcome should look like.
Game Theory Inside Modern AI
- GANs: generative adversarial networks (2014) train two networks against each other, a generator and a discriminator, as a minimax game.
- AI safety debates: some proposals have two AI systems argue opposite sides so a human judge can spot errors.
- Multi-agent AI: as AI agents negotiate, trade and bid for users, mechanism design, the “reverse” of game theory, helps set rules so that honest behavior pays.
- Security: attacker-defender games guide where to place limited defenses, for example in airport patrol scheduling.
Common Misconceptions
“Game theory assumes everyone is selfish.” It assumes players have goals and act on them; goals can include fairness or cooperation.
“A Nash equilibrium is the best outcome for everyone.” Not necessarily: in the prisoner’s dilemma, the equilibrium is worse for both players than cooperating.
“AI wins at poker by reading faces.” Libratus and Pluribus never saw a face; they won with mathematically balanced strategies, including well-timed bluffs.
Watch: Game Theory and AI
If you learn better by watching, these videos are worth your time.
Check Your Understanding
Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.
Answer each question and press Submit to see your result right away.
Frequently Asked Questions
How is game theory used in AI?
To find strong strategies in games with hidden information such as poker, to train models that compete (GANs), to design multi-agent systems and negotiations, and to plan security defenses.
What is a Nash equilibrium?
A set of strategies, one per player, where no player can improve their payoff by changing strategy while the others stay the same. John Nash proved in 1950 that every finite game has one, possibly with randomized strategies.
How did AI beat poker professionals?
Carnegie Mellon’s Libratus (2017) and Pluribus (2019) computed near-equilibrium strategies through self-play and regret minimization, including balanced bluffing, and beat elite players in heads-up and six-player no-limit Texas hold’em.
What is self-play in AI?
Training an AI by having it play against copies of itself, improving its strategy from the results. It powered AlphaZero in chess and Go and the poker AIs.
Why do mixed strategies matter?
When predictability can be exploited, randomizing with the right probabilities is optimal, as in rock-paper-scissors, penalty kicks and poker bluffing.
Keep Exploring
- Reinforcement Learning Explained: Math With Examples
- How AI Predicts the Next Word: Probability Explained
- Can AI Predict the Stock Market? The Math Explained
Sources and Further Reading
- Nash, Equilibrium points in n-person games (PNAS, 1950)
- Brown and Sandholm, Superhuman AI for heads-up no-limit poker: Libratus beats top professionals (Science, 2018)
- Brown and Sandholm, Superhuman AI for multiplayer poker (Science, 2019)
- Chiappori, Levitt and Groseclose, Testing Mixed-Strategy Equilibria When Players Are Heterogeneous: The Case of Penalty Kicks in Soccer (American Economic Review, 2002)
- Meta FAIR Diplomacy Team, Human-level play in the game of Diplomacy by combining language models with strategic reasoning (Science, 2022)
- Goodfellow et al., Generative Adversarial Networks (arXiv 1406.2661)
Related to This Article
More math articles
- How to Fill in the Gaps: Finding Missing Angles of Quadrilaterals
- Kansas Algebra 1 Free Worksheets: 68 Free Printable Algebra 1 Worksheets with Step-by-Step Keys
- Commerce Clause
- Genetic Variation: How to Understand Population Diversity
- The Best Grade 7 ELA Practice Tests for Wisconsin Students
- Introduction to Matrix Inverse
- Roth IRA Calculator (2026 Limits, Growth & Income Eligibility)
- Iron Triangles and Issue Networks
- Digital SAT Math May 2026: What Changed and How to Prep for the Next Test
- Estate Tax Calculator: 2026 Federal Exemption of $15 Million


What people say about "Game Theory and AI: Nash Equilibrium, Poker and Self-Play - Effortless Math"?
No one replied yet.