Multi-Agent Coordination

Discover the latest developments in multi-robot coordination techniques with this insightful and original resource

Multi-Agent Coordination: A Reinforcement Learning Approach delivers a comprehensive, insightful, and unique treatment of the development of multi-robot coordination algorithms with minimal computational burden and reduced storage requirements when compared to traditional algorithms. The accomplished academics, engineers, and authors provide readers with both a high-level introduction to, and overview of, multi-robot coordination, and in-depth analyses of learning-based planning algorithms.

You'll learn about how to accelerate the exploration of the team-goal and alternative approaches to speeding up the convergence of TMAQL by identifying the preferred joint action for the team. The authors also propose novel approaches to consensus Q-learning that address the equilibrium selection problem and a new way of evaluating the threshold value for uniting empires without imposing any significant computation overhead. Finally, the book concludes with an examination of the likely direction of future research in this rapidly developing field.

Readers will discover cutting-edge techniques for multi-agent coordination, including:

• An introduction to multi-agent coordination by reinforcement learning and evolutionary algorithms, including topics like the Nash equilibrium and correlated equilibrium

• Improving convergence speed of multi-agent Q-learning for cooperative task planning

• Consensus Q-learning for multi-agent cooperative planning

• The efficient computing of correlated equilibrium for cooperative q-learning based multi-agent planning

• A modified imperialist competitive algorithm for multi-agent stick-carrying applications

Perfect for academics, engineers, and professionals who regularly work with multi-agent learning algorithms, Multi-Agent Coordination: A Reinforcement Learning Approach also belongs on the bookshelves of anyone with an advanced interest in machine learning and artificial intelligence as it applies to the field of cooperative or competitive robotics.

Autorentext

Arup Kumar Sadhu, PhD, received his doctorate in Multi-Robot Coordination by Reinforcement Learning from Jadavpur University in India in 2017. He works as a scientist with Research & Innovation Labs, Tata Consultancy Services. Amit Konar, PhD, received his doctorate from Jadavpur University, India in 1994. He is Professor with the Department of Electronics and Tele-Communication Engineering at Jadavpur University where he serves as the Founding Coordinator of the M. Tech. program on intelligent automation and robotics.

Inhalt
Preface xi

Acknowledgments xix

About the Authors xxi

1 Introduction: Multi-agent Coordination by Reinforcement Learning and Evolutionary Algorithms 1

1.1 Introduction 2

1.2 Single Agent Planning 4

1.2.1 Terminologies Used in Single Agent Planning 4

1.2.2 Single Agent Search-Based Planning Algorithms 10

1.2.2.1 Dijkstra's Algorithm 10

1.2.2.2 A (A-star) Algorithm 11

1.2.2.3 D (D-star) Algorithm 15

1.2.2.4 Planning by STRIPS-Like Language 15

1.2.3 Single Agent RL 17

1.2.3.1 Multiarmed Bandit Problem 17

1.2.3.2 DP and Bellman Equation 20

1.2.3.3 Correlation Between RL and DP 21

1.2.3.4 Single Agent Q-Learning 21

1.2.3.5 Single Agent Planning Using Q-Learning 24

1.3 Multi-agent Planning and Coordination 25

1.3.1 Terminologies Related to Multi-agent Coordination 25

1.3.2 Classification of MAS 26

1.3.3 Game Theory for Multi-agent Coordination 28

1.3.3.1 Nash Equilibrium 31

1.3.3.2 Correlated Equilibrium 36

1.3.3.3 Static Game Examples 38

1.3.4 Correlation Among RL, DP, and GT 40

1.3.5 Classification of MARL 40

1.3.5.1 Cooperative MARL 42

1.3.5.2 Competitive MARL 56

1.3.5.3 Mixed MARL 59

1.3.6 Coordination and Planning by MAQL 84

1.3.7 Performance Analysis of MAQL and MAQL-Based Coordination 85

1.4 Coordination by Optimization Algorithm 87

1.4.1 PSO Algorithm 88

1.4.2 Firefly Algorithm 91

1.4.2.1 Initialization 92

1.4.2.2 Attraction to Brighter Fireflies 92

1.4.2.3 Movement of Fireflies 93

1.4.3 Imperialist Competitive Algorithm 93

1.4.3.1 Initialization 94

1.4.3.2 Selection of Imperialists and Colonies 95

1.4.3.3 Formation of Empires 95

1.4.3.4 Assimilation of Colonies 96

1.4.3.5 Revolution 96

1.4.3.6 Imperialistic Competition 97

1.4.4 Differential Evolution Algorithm 98

1.4.4.1 Initialization 99

1.4.4.2 Mutation 99

1.4.4.3 Recombination 99

1.4.4.4 Selection 99

1.4.5 Off-line Optimization 99

1.4.6 Performance Analysis of Optimization Algorithms 99

1.4.6.1 Friedman Test 100

1.4.6.2 ImanDavenport Test 100

1.5 Summary 101

References 101

2 Improve Convergence Speed of Multi-Agent Q-Learning for Cooperative Task Planning 111

2.1 Introduction 112

2.2 Literature Review 116

2.3 Preliminaries 118

2.3.1 Single Agent Q-learning 119

2.3.2 Multi-agent Q-learning 119

2.4 Proposed MAQL 123

2.4.1 Two Useful Properties 124

2.5 Proposed FCMQL Algorithms and Their Convergence Analysis 128

2.5.1 Proposed FCMQL Algorithms 129

2.5.2 Convergence Analysis of the Proposed FCMQL Algorithms 130

2.6 FCMQL-Based Cooperative Multi-agent Planning 131

2.7 Experiments and Results 134

2.8 Conclusions 141

2.9 Summary 143

2.A More Details on Experimental Results 144

2.A.1 Additional Details of Experiment 2.1 144

2.A.2 Additional Details of Experiment 2.2 159

2.A.3 Additional Details of Experiment 2.4 161

References 162

3 Consensus Q-Learning for Multi-agent Cooperative Planning 167

3.1 Introduction 167

3.2 Preliminaries 169

3.2.1 Single Agent Q-Learning 169

3.2.2 Equilibrium-Based Multi-agent Q-Learning 170

3.3 Consensus 171

3.4 Proposed CoQL and Planning 173

3.4.1 Consensus Q-Learning 173

3.4.2 Consensus-Based Multi-robot Planning 175

3.5 Experiments and Results 176

3.5.1 Experimental Setup 176

3.5.2 Experiments for CoQL 177 3.5.3 Experiments for Consensus-Based Planning 177</p&...