Author(s): Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel
Venue: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Year Published: 2017
Keywords: visual perception, dynamical systems, neural networks, reinforcement learning
Expert Opinion: The work focuses on one of the most important problems related to utilizing CNNs for robotics problems: transferring policies from simulations to real world. Effective solutions are presented together with promising results.

Author(s): Adam Coates, Pieter Abbeel, Andrew Y. Ng
Venue: International conference on Machine Learning
Year Published: 2008
Keywords: dynamical systems, learning from demonstration
Expert Opinion: An early paper that made learning control popular and showed that ML based control methods can be successfully applied to the real world.

Author(s): Auke Jan Ijspeert, Jun Nakanishi, Stefan Schaal
Venue: IEEE International Conference on Robotics and Automation (ICRA)
Year Published: 2002
Keywords: probabilistic models, nonlinear systems, dynamical systems, learning from demonstration, humanoid robotics
Expert Opinion: In this work, a robust and scaleable movement primitive learning approach is proposed. The key insight is the embedding of motion trajectories in a 2nd order dynamical system. Goal attractors enable the generalization to different targets and simplify the learning of the model parameters from rewards. Complex motion can be learned through least squares regression from demonstrations.

Author(s): Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, Stefan Schaal
Venue: Neural Computation (Volume 25, Issue 2)
Year Published: 2013
Keywords: planning, learning from demonstration, dynamical systems, nonlinear systems
Expert Opinion: The right parametrization is often the key in a learning system. Dynamical movement primitives (Ijspeert, Nakanishi, Schaal, 2003) are a very successful way to encode movements in robots. The idea is to use dynamical systems with desired properties, such as stable attractors or rhythmic solutions, as building blocks. This provides a low-dimensional parametrization and combining them linearly allows for effective learning. So far it was mainly used for learning from demonstration.

Author(s): Stephane Ross, Geoffrey J. Gordon, J. Andrew Bagnell
Venue: 14th International Conference on Artificial Intelligence and Statistics
Year Published: 2011
Keywords: neural networks, learning from demonstration, dynamical systems
Expert Opinion: This paper proposes 'Dagger', an online learning algorithm for sequential decision making problems - a ubiquitous problem especially in robotics. It solves the problem that a policy trained on expert demonstrations may encounter new states that it has not seen in the training data. This will lead to error propagation and finally divergence of the policy. To bound this error, the authors propose to keep aggregating data when rolling out a learned policy and keep retraining the policy on this new data.

Author(s): Bethany R. Leffler, Michael L. Littman, Timothy Edmunds
Venue: AAAI Conference on Artificial Intelligence
Year Published: 2007
Keywords: reinforcement learning, learning from demonstration, dynamical systems
Expert Opinion: This paper from 2007 was the culmination of a thread of reinforcement learning research on relocatable action models. The premise is that states can be clustered into types such that actions taken in states of the same type have similar effects. This paper was the first to study implementation of such an idea on a real robot, and shows impressive results. It's a great example of the creative robot demonstrations from Michael Littman's lab at a time when most learning was limited to simulation.

Author(s): Marc Peter Deisenroth, Dieter Fox, Carl Edward Rasmussen
Venue: IEEE Transactions on Pattern Analysis and Machine Intelligence
Year Published: 2017
Keywords: gaussians, dynamical systems, probabilistic models, reinforcement learning
Expert Opinion: This paper shows the power of model-based reinforcement learning for robot control. It nicely illustrates the power of Gaussian Processes to capture the uncertainty and demonstrates how to leverage it in a highly data-efficient reinforcement learning algorithm. Overall, PILCO (the algorithm described in this paper) might be the most data-efficient algorithm I know. Please note that conference versions of this paper were published at ICML (2011 - PILCO: A model-based and data-efficient approach to policy search) and RSS (2011 - Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning).

Author(s): Aude Billard and Sylvain Calinon, Ruediger Dillmann, Stefan Schaal
Venue: Book
Year Published: 2008
Keywords: humanoid robotics, learning from demonstration, dynamical systems
Expert Opinion: Provides a clear presentation of robot learning from demonstration from the authors who made the approach popular

Author(s): Duy Nguyen-Tuong, Jan Peters
Venue: Cognitive Science
Year Published: 2011
Keywords: gaussians, survey, dynamical systems, optimal control, unsupervised learning, reinforcement learning
Expert Opinion: The only non-RL paper on my list :). Modelling of robots is part of both very classical control approaches as well as modern learning approaches. There are many excellent papers, I chose this one for providing a wide overview. One of my favourite papers on this topic, by the same authors, included in this survey combines insights from analytic modelling (allowing fast identification of a small set of parameters) with Gaussian process modelling (allowing precise and flexible modelling, but at the cost of requiring more data). I chose this survey instead, as it provides a wider overview and is thus something I would be more likely to suggest to a student or mentee to get a wider overview.

Author(s): Robert Platt Jr., Russ Tedrake, Leslie Kaelbling, Tomas Lozano-Perez
Venue: Robotics: Science and Systems VI
Year Published: 2010
Keywords: manipulation, dynamical systems, planning, gaussians
Expert Opinion: This isn't a learning paper, but a planning paper. Nonetheless, I feel the need to include it, as it strongly influenced my thinking about manipulation learning. It was the first work that I had seen make POMDPs work for real robotics problems. It teaches students that reasoning about uncertainty is important, but that you need to make the right assumptions in order to make it work for any reasonably sized problem. Whereas PILCO aims to reduce model uncertainty, this work assumes a correct model and leverages information-gathering actions to reduce state uncertainty. Thus, these papers are complementary when discussing uncertainty.

Author(s): Ashutosh Saxena, Justin Driemeyer, Andrew Y. Ng
Venue: International Journal of Robotics Research
Year Published: 2008
Keywords: neural networks, dynamical systems, visual perception, learning from demonstration, manipulation, planning
Expert Opinion: A key paper in grasp learning. The approach is relatively simple in nature (hack to get grasp orientation, simple features as anchors), but it was at the time a very clear example of an approach that starkly contrasted with mainstream grasping.

Author(s): Emanuel Todorov, Tom Erez and Yuval Tassa
Venue: Software
Year Published: 2012
Keywords: contact dynamics, dynamical systems, reinforcement learning
Expert Opinion: Mujoco (together with Bullet) is probably the most popular simulator for learning based robotics. It's built with robotics research in mind (as opposed to many other simulators that were initially built for games), and consequently has a lot of flexibility around things that matter for robotics, like physical realism, sensors, actuation models, tendons, and so on. It has a clean and easy-to-use interface and is extremely fast, both which are useful features for iterating on new research.

Author(s): Hamid Benbrahim
Venue: University of New Hampshire
Year Published: 1997
Keywords: policy gradients, neural networks, reinforcement learning, dynamical systems, legged robots
Expert Opinion: Totally overlooked work that employs policy gradients with multiple nested CMAC neural networks. One could give Benbrahim credit for having done much of OpenAI's stuff 20 years earlier.

Author(s): Marc Peter Deisenroth, Carl Edward Rasmussen
Venue: International Conference of Machine Learning
Year Published: 2011
Keywords: state estimation, reinforcement learning, probabilistic models, gaussians, dynamical systems, visual perception, policy gradients
Expert Opinion: In principle, model based RL offers many advantages for robot learning, such as efficient use of data and the ability to predict in advance how a trajectory will roll out. In practice, however, getting model based RL to work has proved to be very difficult. In this work, the authors tackle a key difficulty F112 when optimizing a policy for a dynamics model that was learned from data, model errors get exploited by the optimization algorithm. A very elegant solution is proposed: uncertainty estimation should be incorporated into the decision making process, thereby discouraging the optimization to visit states where model uncertainty is high and the predictions are likely to be wrong. This intuitive idea is implemented using Gaussian processes, which offer a principled approach to modeling uncertainty in continuous dynamical systems. The resulting algorithm - PILCO - is demonstrated to be very efficient in sample complexity, improving upon the state of the art by orders of magnitude. This paper introduced several key ideas that have since been implemented in many subsequent works on robot learning and model based RL.

Author(s): Christopher G. Atkeson, Chae H. An, John M. Hollerbach
Venue: International Journal of Robotics Research
Year Published: 1986
Keywords: state estimation, dynamical systems, legged robots
Expert Opinion: For an early example of model based learning.

Author(s): David Vernon
Venue: Book
Year Published: 2014
Keywords: cognitive sciences, dynamical systems, neural networks, humanoid robotics
Expert Opinion: David Vernon's two references highlight the importance of the architecture and the role of its components in learning how to execute and interpret actions. The second is a must-read book for who is interested in the field of artificial cognitive systems.

Author(s): Pieter Abbeel, Adam Coates and Andrew Y. Ng
Venue: International Journal of Robotics Research
Year Published: 2010
Keywords: learning from demonstration, optimal control, dynamical systems
Expert Opinion: Still one of the coolest demos of robot learning.

Author(s): Reza Shadmehr and Ferdinando A. Mussa-lvaldi
Venue: The Journal of Neuroscience
Year Published: 1994
Keywords: dynamical systems, visual perception, planning
Expert Opinion: The reason why I picked these articles and books is because I think that robot learning cannot be separated from the cognitive architecture supporting the learning processes. The first two reference highlight the importance and role of embodiment (in humans and robots) and the fact that in physical systems part of the learning process is embedded in the morphology and material.

Author(s): Jan Peters, Stefan Schaal
Venue: Robotics and Neuroscience
Year Published: 2008
Keywords: manipulation, policy gradients, reinforcement learning, dynamical systems
Expert Opinion: This paper presents some of the early method to learn high-dimensional parameters, which is important in robot manipulation. It presents a non-trivial combination of the authors' earlier work in learning motor skills, that aside from learning how reinforcement learning can be used to learn high-dimensional parameters, it also provide a good short summary of relevant work (up to that point).

Author(s): Karl Sims
Venue: SIGGRAPH
Year Published: 1994
Keywords: dynamical systems, genetic algorithms
Expert Opinion: This paper demonstrated that machine learning and optimization do not have to be restricted to the generation of behavior of a robot. Rather, morphology and shape of an agent can be changed and optimized in an automatic fashion, too. In doing so, the paper created some of the first complex (an extremely impressive and life-like) examples of artificial creatures whose brain and body are fully synthesized. The video accompanying this paper is one of the best research videos out there. The paper has also spawned a number of follow ups, in particular by the group of Hod Lipson at Columbia.