Game Theory and AI: Nash Equilibrium, Poker and Self-Play

Quick answer: AI uses game theory to choose strategies when outcomes depend on other players’ decisions. It models possible actions, calculates expected payoffs and sometimes randomizes its choices so it cannot be predicted. A key concept is the Nash equilibrium, where no player can improve their expected payoff by changing strategy alone. Poker AIs such as Libratus (2017) and Pluribus (2019) used self-play and counterfactual regret minimization to build strong strategies despite hidden information and beat top professionals. Game theory also guides AI in negotiation, security and competition.

Facts checked November 22, 2026.

Chess and Go have perfect information: both players see the whole board. Poker, negotiations, auctions and cybersecurity do not. There, the best move depends on what the other side might do, and the other side is thinking the same about you. The mathematics of these situations is game theory, and it is behind some of AI’s most striking wins.

The Basic Vocabulary

TermMeaning
PlayersThe decision makers
StrategiesThe choices available to each player
PayoffsThe numerical outcome for each combination of choices
Pure strategyAlways making the same choice
Mixed strategyChoosing randomly with set probabilities
Nash equilibriumStrategies where no player can do better by changing alone

John Nash proved in 1950 that every finite game has at least one equilibrium, possibly in mixed strategies. Earlier, in 1928, John von Neumann proved the minimax theorem for two-player zero-sum games: there is a single value each side can guarantee. For a classic two-player example, see the prisoner’s dilemma explained.

Why Randomness Can Be the Best Strategy

In rock-paper-scissors, any predictable pattern can be exploited. The equilibrium is to play each option with probability \(\tfrac{1}{3}\), which gives an expected payoff of 0 against any opponent. Being unpredictable is not a weakness; it is the optimal strategy. Poker bluffing works the same way: a player who never bluffs is easy to read.

Worked Example: The Penalty Kick

A kicker chooses left (L) or right (R); the goalkeeper dives left or right. The table shows the chance of a goal:

Goalie dives LGoalie dives R
Kick L0.50.9
Kick R0.80.4

Let \(p\) be the probability the kicker kicks left and \(q\) the probability the goalie dives left. Neither pure choice is safe, so both mix. At equilibrium, each player randomizes so the other is indifferent. If the goalie dives left with probability \(q\), the kicker’s scoring chances are equal when

\[ 0.5q + 0.9(1 – q) = 0.8q + 0.4(1 – q) \;\Rightarrow\; 0.9 – 0.4q = 0.4 + 0.4q \;\Rightarrow\; q = 0.625. \]

If the kicker kicks left with probability \(p\), the goalie is indifferent when

\[ 0.5p + 0.8(1 – p) = 0.9p + 0.4(1 – p) \;\Rightarrow\; 0.8 – 0.3p = 0.4 + 0.5p \;\Rightarrow\; p = 0.5. \]

The goal probability at equilibrium is \(0.5 \times 0.5 + 0.8 \times 0.5 = 0.65\), whichever way the goalie dives. Neither side can raise its result by changing alone. These numbers are hypothetical, but a 2002 study by Pierre-André Chiappori, Steven Levitt and Timothy Groseclose found real professional penalty kicks consistent with several mixed-strategy predictions. Brush up on probability if the expected values are new.

How AI Bluffs: The Math of a Balanced Bet

On the last betting round of a poker hand, suppose the pot holds 100 chips and a player bets 100. The opponent must call 100 to win 200 (the pot plus the bet). If a fraction \(b\) of the player’s bets are bluffs, calling gains 200 against a bluff and loses 100 against a strong hand. The caller is indifferent when

\[ 200b = 100(1 – b) \quad \Rightarrow \quad b = \tfrac{1}{3}. \]

So a balanced strategy bluffs about one bet in three in this spot. Bluff more and the opponent should always call; bluff less and they should always fold. Poker AIs compute balances like this across millions of situations. A strategy’s exploitability measures how much a perfect opponent could win against it; strong poker AIs aim to keep it near zero.

How AI Finds Equilibria: Self-Play

Real games are far too large to solve by hand: heads-up no-limit Texas hold’em has an astronomically large number of possible betting situations, so even computers must use simplified versions of the game. AI systems find good strategies by playing against copies of themselves millions of times and adjusting. A key algorithm, counterfactual regret minimization (CFR), tracks for each decision how much better each alternative would have done (“regret”) and shifts probability toward the choices it regrets not making. In two-player zero-sum games, the average strategy provably approaches a Nash equilibrium; in games with more players, self-play gives no such guarantee, though it often works well in practice. This is a cousin of reinforcement learning.

Milestones: Game Theory Meets AI

YearSystemAchievement
2015Cepheus (University of Alberta)Essentially (weakly) solved heads-up limit Texas hold’em
January 2017Libratus (Carnegie Mellon)Beat four top professionals over 120,000 hands of heads-up no-limit hold’em
2019Pluribus (Carnegie Mellon and Facebook AI)Beat elite professionals in six-player no-limit hold’em, where Nash guarantees no longer apply cleanly
2022Cicero (Meta)Ranked in the top 10% of online Diplomacy participants who played more than one game, negotiating in natural language

Pluribus is notable because six-player poker is not zero-sum between two sides, so the clean equilibrium guarantees break down; self-play strategies still proved strong in practice. Cicero combined a language model with strategic planning to negotiate, make deals and keep promises with human players.

Game Theory vs Reinforcement Learning

Reinforcement learning is about one agent learning from rewards; game theory is about several decision makers whose rewards depend on each other. They meet in self-play: each copy of the AI is learning by reinforcement while the others form its changing environment, and game theory tells us what a stable outcome should look like.

Game Theory Inside Modern AI

  • GANs: generative adversarial networks (2014) train two networks against each other, a generator and a discriminator, as a minimax game.
  • AI safety debates: some proposals have two AI systems argue opposite sides so a human judge can spot errors.
  • Multi-agent AI: as AI agents negotiate, trade and bid for users, mechanism design, the “reverse” of game theory, helps set rules so that honest behavior pays.
  • Security: attacker-defender games guide where to place limited defenses, for example in airport patrol scheduling.

Common Misconceptions

“Game theory assumes everyone is selfish.” It assumes players have goals and act on them; goals can include fairness or cooperation.

“A Nash equilibrium is the best outcome for everyone.” Not necessarily: in the prisoner’s dilemma, the equilibrium is worse for both players than cooperating.

“AI wins at poker by reading faces.” Libratus and Pluribus never saw a face; they won with mathematically balanced strategies, including well-timed bluffs.

Watch: Game Theory and AI

If you learn better by watching, these videos are worth your time.

Nash Equilibrium in 5 Minutes (Ashley Hodgson). A clear introduction to Nash equilibrium with payoff tables.
Poker-Playing AI Beats Pro Players (Carnegie Mellon University). Carnegie Mellon researchers on how their poker AI was built.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. What is the equilibrium strategy in rock-paper-scissors?
Found a bug? Let us know!
2. In the penalty example, with what probability should the goalie dive left?
Found a bug? Let us know!
3. Pot 100, bet 100: what fraction of bets should be bluffs to make calling break even?
Found a bug? Let us know!
4. What does a Nash equilibrium mean?
Found a bug? Let us know!
5. How did poker AIs such as Libratus learn?
Found a bug? Let us know!

Frequently Asked Questions

How is game theory used in AI?

To find strong strategies in games with hidden information such as poker, to train models that compete (GANs), to design multi-agent systems and negotiations, and to plan security defenses.

What is a Nash equilibrium?

A set of strategies, one per player, where no player can improve their payoff by changing strategy while the others stay the same. John Nash proved in 1950 that every finite game has one, possibly with randomized strategies.

How did AI beat poker professionals?

Carnegie Mellon’s Libratus (2017) and Pluribus (2019) computed near-equilibrium strategies through self-play and regret minimization, including balanced bluffing, and beat elite players in heads-up and six-player no-limit Texas hold’em.

What is self-play in AI?

Training an AI by having it play against copies of itself, improving its strategy from the results. It powered AlphaZero in chess and Go and the poker AIs.

Why do mixed strategies matter?

When predictability can be exploited, randomizing with the right probabilities is optimal, as in rock-paper-scissors, penalty kicks and poker bluffing.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "Game Theory and AI: Nash Equilibrium, Poker and Self-Play - Effortless Math"?

No one replied yet.

Leave a Reply