Author(s): R. E. Kalman
Venue: Transactions of the ASME–Journal of Basic Engineering
Year Published: 1960
Keywords: probabilistic models, optimal control, dynamical systems, state estimation
Expert Opinion: The paper that introduced the Kalman Filter: probably the most used inference algorithm in science and engineering. The paper *also* introduced the linear quadratic regulator as a bonus: probably the most used optimal control algorithm. It is clearly written and easy to understand.

Author(s): Tamar Flash, Neville Hogans
Venue: Journal of Neuroscience
Year Published: 1985
Keywords: optimal control, cognitive sciences, dynamical systems
Expert Opinion: This paper is part of a set of papers that outlines important points about synergy formation in neuroscience and robotics along a fil-rouge during the last 40 years: a) the coordination of multiple joints in goal directed movement, b) the characterization of "biological motion‚", c) the equilibrium point hypothesis, d) the role of force fields for motor coordination, e) the extension of the equilibrium point hypothesis from real (overt) movements to covert (real) movements, f) the characterization of synergy formation as the simulation of an internal body model. In particular, this paper explained the bell-shape of the speed profile in human arm reaching movements in terms of optimal control. The observation had been first made in Morasso P (1981) Spatial control of arm movements. Experimental Brain Research.

Author(s): Christopher G. Atkeson, Chae H. An, John M. Hollerbach
Venue: International Journal of Robotics Research
Year Published: 1986
Keywords: state estimation, dynamical systems, legged robots
Expert Opinion: For an early example of model based learning.

Author(s): Valentino Braitenberg
Venue: MIT Press
Year Published: 1986
Keywords: cognitive sciences
Expert Opinion: This book gave rise to the concept of Braitenberg Vehicles. It follows a sequence of simple thought experiments, starting with the idea that light sensors attached to motors can generate a machine that can simulate a desire for or aversion to light. These concepts are made progressively more complex, ranging up to machines that can display sophisticated behaviours. This is an interesting foundational take on emergent intelligence.

Author(s): J.J. Slotine and W. Li
Venue: International Journal of Robotics Research
Year Published: 1987
Keywords: dynamical systems, manipulation
Expert Opinion: I selected this paper for several reasons: 1) to remind us that learning system dynamics at the same time as performing control is an old control topic, known as adaptive control, and that fundamental results in adaptive control and especially fundamental limitations will certainly carry over in other learning problems (parametric or not) when learning and control time-scales are mixed, 2) it is an incredibly elegant paper and its results have been further used on real robotic systems to provide fast adaptation to changing dynamics (e.g. plane catching experiments done at MIT using adaptive control in the early 90s) and they have also been extended to use non-parametric models (e.g. neural networks and Gaussian radial basis function) in subsequent papers.

Author(s): Dean A. Pomerleau
Venue: MITP
Year Published: 1989
Keywords: mobile robots, learning from demonstration, neural networks
Expert Opinion: This work was pioneering with respect to machine learning in robotics broadly, learning from demonstration specifically, and also autonomous driving. It applied a neural net to learn steering angles from examples of human driving (even online!), way back in 1989. By today's deep learning standards the net was tiny (5 hidden units) and the sensor input extremely limited (30x32 image pixels), but it worked... and at a time when robots rarely operated outside of the lab or factory, and machine learning was rarely deployed on real hardware. It is the first* example of using demonstration-based learning for high-stakes control, that required (comparatively) fast sampling (25Hz) and operated a large van at regular road speeds (20mph). The vehicle was a part of NavLab, which was the precursor to CMU's DARPA Grand and Urban Challenges entires in the early 2000's, and those challenges in turn played a big role in accelerating today's driverless car boom. * To my knowledge! ...also, there actually are two papers (and my words above mix the two): [first publication] D. Pomerleau. ALVINN: An Autonomous Land Vehicle in a Neural Network. In Advances in Neural Information Processing Systems, 1989. [real world driving results] D. Pomerleau. Efficient Training of Artificial Neural Networks for Autonomous Navigation. Neural Compuation, 3(1), 88-97, 1991.

Author(s): Christopher G. Atkeson
Venue: Advances in Neural Information Processing Systems
Year Published: 1990
Keywords: cognitive sciences
Expert Opinion: seminal work on using learned local linear models for robot learning

Author(s): Sridhar Mahadevan and Jonathan Connell
Venue: Artificial Intelligence
Year Published: 1991
Keywords: reinforcement learning
Expert Opinion: This is a seminal paper that really broke open the application of RL to learning policies. It was the first that I am aware of, and certainly the earliest high-profile case. It blew a lot of minds when it came out.

Author(s): Michael I. Jordan, David E. Rumelhart
Venue: Cognitive Science
Year Published: 1992
Keywords: dynamical systems, neural networks
Expert Opinion: explains that inverse model learning may be flawed, and one way how to circumvent this flaw

Author(s): Andrew W. Moore, Christopher G. Atkeson
Venue: Machine Learning Journal
Year Published: 1993
Keywords: reinforcement learning, neural networks
Expert Opinion: Quintessential technical insights, super important up to today.

Author(s): Minoru Asada, Eiji Uchibe, Shoichi Noda, Sukoya Tawaratsumida, Koh Hosoda
Venue: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Year Published: 1994
Keywords: reinforcement learning, dynamical systems
Expert Opinion: This paper is representative of a body of work from Minoru Asada's lab in the mid 1990's that was some of the first examples of machine learning in the robot soccer domain, some of the first reinforcement learning on real robots, some of the first RL from visual inputs, and some of the first multirobot learning, all wrapped into one. They introduced several methods for aggressively reducing the sample complexity needed for learning on real robots, such as "learning from easy missions."

Author(s): Craig W. Reynolds
Venue: International Conference on Simulation of Adaptive Behavior
Year Published: 1994
Keywords: genetic algorithms, evolution
Expert Opinion: The work features the automatic synthesis of a symbolic robot controller in a non-deterministic environment via genetic programming. Despite being an early paper on robot learning it features a combination of many aspects that are often not found in modern papers, i.e., (1) learning of explainable, symbolic code, (2) automatic sensor placement, (3) strong non-determinism. Reynolds even goes to great lengths to analyse the code generated by the evolutionary process and identifies a more general framework for how a good solution looks like. Structure and interpretability play an important role in this paper.

Author(s): Karl Sims
Venue: SIGGRAPH
Year Published: 1994
Keywords: dynamical systems, genetic algorithms
Expert Opinion: This paper demonstrated that machine learning and optimization do not have to be restricted to the generation of behavior of a robot. Rather, morphology and shape of an agent can be changed and optimized in an automatic fashion, too. In doing so, the paper created some of the first complex (an extremely impressive and life-like) examples of artificial creatures whose brain and body are fully synthesized. The video accompanying this paper is one of the best research videos out there. The paper has also spawned a number of follow ups, in particular by the group of Hod Lipson at Columbia.

Author(s): Reza Shadmehr and Ferdinando A. Mussa-lvaldi
Venue: The Journal of Neuroscience
Year Published: 1994
Keywords: dynamical systems, visual perception, planning
Expert Opinion: The reason why I picked these articles and books is because I think that robot learning cannot be separated from the cognitive architecture supporting the learning processes. The first two reference highlight the importance and role of embodiment (in humans and robots) and the fact that in physical systems part of the learning process is embedded in the morphology and material.

Author(s): Christopher G. Atkeson
Venue: Neural Information Processing Systems
Year Published: 1994
Keywords: trajectory optimization, optimal control, dynamic programming
Expert Opinion: A forerunner to guided policy search: introduces the idea of using trajectory optimizers to speed up policy learning.

Author(s): Robert Stengel
Venue: Book
Year Published: 1994
Keywords: optimal control, state estimation
Expert Opinion: Robotics Learning Practitioners must be aware of and understand Optimal Control. :)

Author(s): Gillian Hayes, Yiannis Demiris
Venue: Neural Information Processing Systems Conference (NeurIPS)
Year Published: 1995
Keywords: reinforcement learning, learning from demonstration, dynamical systems
Expert Opinion: This paper introduced learning from imitation. This has proved useful in and of its own (i.e., as a means for non-expert users to program robots), and also as the means for initializing robot controllers (most prominently by Schaal and later Peters) to a reasonable policy that is later refined by learning. Schaal's work on this was probably more influential but it was preceded, and possibly inspired by, Gillian Hayes.

Author(s): Leslie Pack Kaelbling, Michael L. Littman, Andrew W. Moore
Venue: Journal of Artificial Intelligence Research
Year Published: 1996
Keywords: neural networks, survey, reinforcement learning, probabilistic models
Expert Opinion: This work provides a relatively short and easy to understand introduction to Reinforcement Learning. Although rather old and therefore does not cover the new approaches to reinforcement learning, it covers the problem of RL very well. I usually ask beginning students interested in reinforcement learning to read this paper together with the more recent "Reinforcement Learning in Robotics: A Survey" by Jens Kober, Andrew Bagnell, and Jan Peters, and deep learning approaches to reinforcement learning.

Author(s): Yasuo Kuniyoshi, Masayuki Inaba, Hirochika Inoue
Venue: IEEE Transactions on Robotics and Automation
Year Published: 1996
Keywords: learning from demonstration
Expert Opinion: One of the most significant works on Robot Imitation Learning. This set the stages for all works followed.

Author(s): Marcos Salganicoff, Lyle H. Ungar, Ruzena Bajcsy
Venue: Machine Learning, 23, 251-278
Year Published: 1996
Keywords: manipulation
Expert Opinion: This work as far as I can detect is the first introducing Active learning and Forgetting for the Perception-Action paradigm. It was the PhD thesis of Marcos Salganicoff in 1992. The paper (in fact several papers) cited here was published later. The new approach allowed the learner to control when and where in the input space the new examples should be gathered. It balances the cost of gathering the experiences (Exploration) with the cost of misclassification and the execution of the task (Exploitation ).