SequentialSocialDilemmas.PDF

Multi-agent Reinforcement Learning in Sequential Social Dilemmas Joel Z. Leibo1 Vinicius Zambaldi1 Marc Lanctot DeepMind, London, UK DeepMind, London, UK DeepMind, London, UK jzl@ vzambaldi@ lanctot@ Janusz Marecki Thore Graepel DeepMind, London, UK DeepMind, London, UK tartel@ thore@ ABSTRACT The theory of repeated general-sum matrix games pro- Matrix games like Prisoner’s Dilemma have guided research vides a framework for understanding social dilemmas. Fig. on social dilemmas for decades. However, they necessarily 1 shows payo? matrices for three canonical examples: Pris- treat the choice to cooperate or defect as an atomic action. oner’s Dilemma, Chicken, and Stag Hunt. The two actions In real-world social dilemmas these choices are temporally are interpreted as cooperate and defect respectively. The extended. Cooperativeness is a property that applies to poli- four possible outcomes of each stage game are R (reward cies, not elementary actions. We introduce sequential social of mutual cooperation), P (punishment arising from mu- dilemmas that share the mixed incentive structure of matrix tual defection), S (sucker outcome obtained by the player game social dilemmas but also require agents to learn poli- who cooperates with a defecting partner), and T (tempta- cies that implement their strategic intentions. We analyze tion outcome achieved by defecting against a cooperator). the dynamics of policies learned by multiple self-interested A matrix game is a social dilemma when its four payo?s independent learning agents, ea