Author(s): Lerrel Pinto, Abhinav Gupta
Venue: IEEE International Conference on Robotics and Automation (ICRA)
Year Published: 2015
Keywords: manipulation, reinforcement learning, neural networks
Expert Opinion: This paper demonstrated that it's possible to have a robot interact in a self-supervised way with the environment in order to learn useful tasks, like grasping. By running a robot for a long period of time, it's possible to collect enough data to train policies using simple algorithms. This lead the way for a lot of follow up work from Google and others, and is likely an area where we'll see a lot of interest in the future.

Author(s): Nate Kohl, Peter Stone
Venue: IEEE International Conference on Robotics and Automation (ICRA)
Year Published: 2004
Keywords: reinforcement learning, policy gradients, locomotion, legged robots
Expert Opinion: The paper is one of the first impressive applications of policy gradient algorithms on real robots. The policy gradient algorithm is rather simple, but is able to optimize the gait of the AIBO robot efficiently.

Author(s): Pieter Abbeel, Andrew Y. Ng
Venue: International Conference on Machine Learning
Year Published: 2004
Keywords: reinforcement learning, learning from demonstration
Expert Opinion: Provided a convincing demonstration of the usefulness of inverse reinforcement learning

Author(s): George Konidaris, Leslie Pack Kaelbling, Tomas Lozano-Perez
Venue: Journal of Artificial Intelligence Research
Year Published: 2018
Keywords: probabilistic models, planning
Expert Opinion: Abstraction is an important aspect of robot learning. This paper addresses the issue of learning state abstractions for efficient high-level planning. Importantly, the state abstraction should be induced from the set of skills/options that the robot is capable of executing. The resulting abstraction can then be used to determine if any plan is feasible. The paper addresses both deterministic and probabilistic planning. It is also a great example of learning the preconditions and effects of skills for planning complex tasks.

Author(s): Marc Peter Deisenroth, Gerhard Neumann, Jan Peters
Venue: Book
Year Published: 2013
Keywords: survey, reinforcement learning
Expert Opinion: A great unifying view on policy search

Author(s): Dean A. Pomerleau
Venue: MITP
Year Published: 1989
Keywords: mobile robots, learning from demonstration, neural networks
Expert Opinion: This work was pioneering with respect to machine learning in robotics broadly, learning from demonstration specifically, and also autonomous driving. It applied a neural net to learn steering angles from examples of human driving (even online!), way back in 1989. By today's deep learning standards the net was tiny (5 hidden units) and the sensor input extremely limited (30x32 image pixels), but it worked... and at a time when robots rarely operated outside of the lab or factory, and machine learning was rarely deployed on real hardware. It is the first* example of using demonstration-based learning for high-stakes control, that required (comparatively) fast sampling (25Hz) and operated a large van at regular road speeds (20mph). The vehicle was a part of NavLab, which was the precursor to CMU's DARPA Grand and Urban Challenges entires in the early 2000's, and those challenges in turn played a big role in accelerating today's driverless car boom. * To my knowledge! ...also, there actually are two papers (and my words above mix the two): [first publication] D. Pomerleau. ALVINN: An Autonomous Land Vehicle in a Neural Network. In Advances in Neural Information Processing Systems, 1989. [real world driving results] D. Pomerleau. Efficient Training of Artificial Neural Networks for Autonomous Navigation. Neural Compuation, 3(1), 88-97, 1991.

Author(s): Marc Peter Deisenroth, Carl Edward Rasmussen
Venue: International Conference of Machine Learning
Year Published: 2011
Keywords: state estimation, reinforcement learning, probabilistic models, gaussians, dynamical systems, visual perception, policy gradients
Expert Opinion: This paper showed in an impressive way how to leverage modern probabilistic methods and model-based reinforcement learning to enable fast policy search. It has become THE reference modeling and inference in nondeterministic tasks. The authors use analytical gradients for efficient policy updates, thereby eschewing the typical problems related to sampling methods. The result is an approach that can learn the cart-pole swing up on a real device in about 20 seconds. If you are doing anything related to reinforcement learning with probabilistic methods, this is a must-read.

Author(s): Brian D. Ziebart, Andrew Maas, J.Andrew Bagnell, and Anind K. Dey
Venue: AAAI Conference on Artificial Intelligence
Year Published: 2008
Keywords: probabilistic models, learning from demonstration, reinforcement learning
Expert Opinion: This work provides a novel and useful approach to the problem of inverse reinforcement learning. It is commonly used in practice and has influenced many follow up works in modeling humans in human-robot interaction.

Author(s): Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba
Venue: Neural Information Processing Systems Conference (NeurIPS)
Year Published: 2018
Keywords: manipulation, humanoid robotics, reinforcement learning, neural networks
Expert Opinion: A really nice, simple idea for learning parameterized skills (building on UVFAs) and efficiently dealing with sparse reward. I think Learning Parameterized Motor Skills on a Humanoid Robot (Castro Da Silva et. al) has a much better description of the parameterized skill learning problem than the HER or UVFA papers, but the HER paper has better practical ideas.

Author(s): Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal
Venue: IEEE International Conference on Robotics and Automation (ICRA)
Year Published: 2009
Keywords: planning, learning from demonstration
Expert Opinion: Not the first DMP paper, but the most understandable and with fixes to some annoying problems with the original formulation. Incredibly simple idea, but that's the nice thing about it -- it is a great starting point for talking about what generalization means in policy learning and how a restricted policy representation with the right inductive bias can allow you to learn something meaningful from a single trajectory, as well as learn quickly from practice.

Author(s): Alexandros Paraschos, Christian Daniel, Jan Peters, and Gerhard Neumann
Venue: Neural Information Processing Systems Conference (NeurIPS)
Year Published: 2013
Keywords: manipulation, probabilistic models, gaussians, planning, learning from demonstration
Expert Opinion: This and the following papers using ProMPs, because they provided a very nice formulation for representing probabilistic movement primitives. ProMPs have many advantages and I found them better than classical DMPs in many robotics applications, from gestures to whole-body manipulations.

Author(s): Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, Stefan Schaal
Venue: Neural Computation (Volume 25, Issue 2)
Year Published: 2013
Keywords: planning, learning from demonstration, dynamical systems, nonlinear systems
Expert Opinion: DMPs proved to be a very useful representation for robot learning. The paper gives the clearest presentation ten years after they were invented

Author(s): Stephane Ross, Geoffrey J. Gordon, J. Andrew Bagnell
Venue: 14th International Conference on Artificial Intelligence and Statistics
Year Published: 2011
Keywords: neural networks, learning from demonstration, dynamical systems
Expert Opinion: Introduces Data Aggregation and the general approach of viewing policy optimization as online learning. Formalizes the notion of interaction with an expert as surrogate objective to the usual policy optimization objective.

Author(s): Pierre-Yves Oudeyer, Frederic Kaplan, and Verena V. Hafner
Venue: IEEE Transactions on Evolutionary Computation (Volume 11, Issue 2)
Year Published: 2007
Keywords: reinforcement learning, evolution, neural networks
Expert Opinion: This article describes some of the first successful experiments about "curious robots" and intrinsic motivation. It is one of the foundational articles in the "developmental robotics" field and inspired hundreds of papers about intrinsic motivation in reinforcement learning.

Author(s): Ashutosh Saxena, Justin Driemeyer, Andrew Y. Ng
Venue: International Journal of Robotics Research
Year Published: 2008
Keywords: neural networks, dynamical systems, visual perception, learning from demonstration, manipulation, planning
Expert Opinion: One of the first papers using general visual features for grasping

Author(s): Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel
Venue: Journal of Machine Learning Research
Year Published: 2016
Keywords: manipulation, probabilistic models, planning, locomotion, learning from demonstration, reinforcement learning, neural networks, visual perception
Expert Opinion: This work has drawn attention to end-to-end learning with neural networks, which I think was the beginning of the big boom of the deep learning in robotics.

Author(s): Sebastian Thrun, Wolfram Burgard, Dieter Fox
Venue: Book
Year Published: 2005
Keywords: probabilistic models
Expert Opinion: It laid out basis for robotics in uncertain real world.

Author(s): Pieter Abbeel, Adam Coates and Andrew Y. Ng
Venue: International Journal of Robotics Research
Year Published: 2010
Keywords: learning from demonstration, optimal control, dynamical systems
Expert Opinion: This paper presents a beautiful and compelling demonstration of the strength of learning dynamical models and using optimal control to learn complex tasks on intrinsically unstable systems even if the learned models rather crude and the optimal controllers are based on linearization, both strong approximations of reality. Furthermore, it addresses the problem of learning from demonstrations and improving from such demonstrations to beat human performance. To the best of my knowledge, on of the first paper demonstrating the use of learning by demonstration, model learning and optimal control together to achieve acrobatic tasks.

Author(s): Auke Jan Ijspeert, Jun Nakanishi, Stefan Schaal
Venue: IEEE International Conference on Robotics and Automation (ICRA)
Year Published: 2002
Keywords: probabilistic models, nonlinear systems, dynamical systems, learning from demonstration, humanoid robotics
Expert Opinion: First work that proproses practical movement primitive representation for robotics. Very concise paper: shows how much can be packed into 6 pages.

Author(s): Richard S. Sutton and Andrew G. Barto
Venue: Book
Year Published: 2018
Keywords: mobile robots, reinforcement learning, unsupervised learning, optimal control, genetic algorithms
Expert Opinion: It presents the definite theoretical basis of reinforcement learning, used widely in robotics.