Author(s): Hansen, Nikolaus, Sibylle D. MYller, and Petros Koumoutsakos.
Venue: Evolutionary Computation
Year Published: 2003
Keywords: evolution
Expert Opinion: This work proposes one of the most efficient black box optimizers based on 2nd order stochastic optimization. The strengths of this approach are that it only has one hyper parameter, the initial learning rate, and that it is simple which has resulted in numerous open access implementations in various programming languages.

Author(s): Freek Stulp, Olivier Sigaud
Venue: Paladyn
Year Published: 2013
Keywords: reinforcement learning, evolution
Expert Opinion: Synthesizing and unifying several generations of work using RL for robotics, converging to algorithms that are identifiable with evolution strategies coming from the evolutionary computation literature.

Author(s): Craig W. Reynolds
Venue: International Conference on Simulation of Adaptive Behavior
Year Published: 1994
Keywords: genetic algorithms, evolution
Expert Opinion: The work features the automatic synthesis of a symbolic robot controller in a non-deterministic environment via genetic programming. Despite being an early paper on robot learning it features a combination of many aspects that are often not found in modern papers, i.e., (1) learning of explainable, symbolic code, (2) automatic sensor placement, (3) strong non-determinism. Reynolds even goes to great lengths to analyse the code generated by the evolutionary process and identifies a more general framework for how a good solution looks like. Structure and interpretability play an important role in this paper.

Author(s): Joel Lehman and Kenneth O. Stanley
Venue: Evolutionary Computation
Year Published: 2011
Keywords: evolution, neural networks, locomotion
Expert Opinion: This works nicely demonstrates that optimizing a reward function is not necessarily the best way to find a solution in a complex search space (especially when the search space is deceptive). It proposes to replace the reward function by a behavioral novelty score, which echoes many of the work in developmental robotics. The experiments described in this paper led to an inspirational book (Why greatness cannot be planned: The myth of the objective, Springer, 2015).

Author(s): Emanuel Todorov
Venue: Nature Neuroscience
Year Published: 2004
Keywords: evolution, learning from demonstration, optimal control, dynamical systems
Expert Opinion: From the paper's abstract: "The sensorimotor system is a product of evolution, development, learning and adaptation-which work on different time scales to improve behavioral performance. Consequently, many theories of motor function are based on 'optimal performance': they quantify task goals as cost functions, and apply the sophisticated tools of optimal control theory to obtain detailed behavioral predictions. The resulting models, although not without limitations, have explained more empirical phenomena than any other class.‚" This paper provides a solid theoretical perspective on how to think about control principally in terms of objectives. It makes a very good case for sensory feedback, utilizing which is a key aspect of robot learning works.

Author(s): Hod Lipson, Jordan B. Pollack
Venue: Nature
Year Published: 2000
Keywords: evolution, neural networks
Expert Opinion: This paper describes the first "complete" autonomous robotics design experiment, from the design to the 3D printing. Technically, the paper leverages all the work from the 1990's in evolutionary robotics, combined with 3D printing (recent at that time), and demonstrates that a machine can design a machine. This paper inspired numerous researchers.

Author(s): Matthias Hebbel, Ralf Kosse and Walter Nistico
Venue: IEEE-RAS International Conference of Humanoid Robots
Year Published: 2006
Keywords: locomotion, legged robots, genetic algorithms, evolution
Expert Opinion: This paper describes the open loop modelling of a robot gait which mimics the human walking style. The authors develop a parameterized model for the leg and arm motions. They then compare various machine learning methods for finding the best parameters, i.e., the ones that provide the best walk. The paper is very interesting as: - it is one of the pioneer works on robot gait learning - it rises many issues related to practical application of machine learning methods on real hardware - it gives many insights (again, mainly practical) on how to develop a robot learning framework. While the scientific contribution may be limited, the paper has a great importance for its presentation of practical issues. For this reason, I recommend its reading to young students interested in studying this topic for the first time.

Author(s): Pierre-Yves Oudeyer, Frederic Kaplan, and Verena V. Hafner
Venue: IEEE Transactions on Evolutionary Computation (Volume 11, Issue 2)
Year Published: 2007
Keywords: reinforcement learning, evolution, neural networks
Expert Opinion: This paper proposes exploration algorithms based on the idea of intrinsic motivations, in particular motivations to explore in order to maximise the learning progress of a robot. This is a prominent example of the work of the Developmental Robotics community that ties link between developmental psychology, neurosciences and concrete robotics implementation and shows that exploring with this approach to learn to predict action consequences (forward models) results in behavior that is organized and shows similarity with human behavior.

Author(s): Sonia Chernova, Manuela Veloso
Venue: International Conference on Intelligent Robots and Systems
Year Published: 2004
Keywords: planning, mobile robots, evolution, legged robots, genetic algorithms, locomotion
Expert Opinion: This paper presents a clear and concrete mapping of genetic algorithms to a compelling hardware domain: Sony AIBO walking gait and the RoboCup soccer competition. The AIBO was an example of a platform where parameter tuning by hand is particularly tedious (54 parameters), plus the platform was safe to have "practice‚" on its own overnight (i.e. without human supervision, a rarity for mobile robots)---offering an opportunity for fully autonomous and on-hardware optimization-based learning, where it was feasible for each generation to be evaluated (according to the fitness function) on the actual robot platform without human supervision or intervention. The learned walk that resulted outperformed all hand-tuned and learned walks that participated in (including that which won) the RoboCup 2003 competition. ** This recommendation is getting a bit into the weeds of specific algorithms---not quite sure if the list is planning to go that deep. It's a work I would present in a class, as a great example of a CS algorithm being translated for use on real robot hardware. Again, not quite sure if that sort of categorization fits the bill.