Emre Ugur


Special Topics in CMPE: Robot Learning (CMPE58Y)
Wednesday 14:00-17:00, BM A5

This course is about (i) "general" approaches that aim development of robot intelligence, and (ii) "more focused" advanced learning methods that endow robots with a particular set of sensorimotor skills. To get a grasp of general approaches, we study the general framework of developmental robotics, active learning and intrinsic motivation, bottom-up skill development and symbol acquisition. These approaches are formulated mostly in an interdisciplinary manner, in relation to the findings from infant development, human information processing, experimental and ecological psychology. For the latter one, we will focus on particular methods that are effective in learning manipulation skills and sensorimotor representations such as learning by demonstration, grasp learning, and probabilistic modeling. Rather than detailed analysis of the Machine Learning methods, we will focus on their exploitation for different robot learning problems.

Schedule (Tentative):

10.02 Course Overview, Developmental Robotics, Intrinsic Motivation I.
Assignment 1
17.02 Perceptual Learning, Motor Learning, Affordances I.
Assignment 2
24.02 No class
02.03 Notes, Reinforcement Learning, MDP, Dynamic Programming, Monte Carlo, TD Learning, ROS tutorial notes I.
Assignment 3
09.03 Notes, TD Learning, Eligibility traces, Function approx., Metehan: Mnih et al., Yigit: Konidaris et al. I. + S.
Programming Assignment
Guest Talk: Assist. Prof. Barkan Ugurlu, Ozyegin University,
Learning to Walk Efficiently: Experiments on the CoMAn Humanoid

Cihan: Kober et al.
S. + G.
Assignment 4
23.03 Yasemin: Argall et al., Ersin: Dynamic Movement Primitives, Intro to Grasping S. + I.
30.03 Ilke: Calinon et al., project proposals S.
Assignment 5
06.04 GMM-Scaffolding-Jacobian Secil:PI^2, I. + S.
Assignment 6
Guest Talk: Assoc. Prof. Erhan Oztop, Ozyegin University (16:00-17:00)
Robot skill generation through Human Sensorimotor Learning
Yigit: Inverse Reinforcement Learning, Okan: Saxena et al. Ersin: Detry et al.
G. + S.
20.04 Spring break
25.04 Midterm G.+ S.
04.05 Probabilistic Approaches S.
11.05 Skill/symbol acquisition S.
26.05 Finals - project presentations S.

Instructor (I.), Student (S.), Guest (G.)

Expected Outcome:
  • Breath: Through introductory material presented by the instructor, and paper presentations by students: An overview of the state-of-the-art methods in robot learning, particularly in developmental robotics, learning by demonstration, policy search methods and reinforcement learning, probabilistic modeling of sensorimotor experience, grasp synthesis algorithms.
  • Depth: Through implementing and extending one of the advanced methods/papers in the final project, an in-depth knowledge on the corresponding method and the robot learning task.

Format: This course follows a mixed lecture-seminar format.
  • Basics and concepts are introduced by the instructor.
  • Research papers are presented by students.
  • Written reports and in-class discussions on the presented papers are essential for the "breath".
  • Project implementation is essential for the "depth".

  • Project: 35%
  • Presentations: 15%
  • Midterm: 15%
  • Reading assignments: 15%
  • Quizzes: 10%
  • In-class activity: 5%
  • Programming assignment: 5%

Final Project: Students might suggest implementing a completely novel approach or choose replicating and extending one high-impact paper. A final project report written in conference/workshop paper format is expected. The topics are not limited to the ones discussed in the lectures as long as they are related to robot learning.
Tentative reading material (to be extended)
Developmental robotics:
  • Asada, M., MacDorman, K. F., Ishiguro, H., and Kuniyoshi, Y. (2001). Cognitive developmental robotics as a new paradigm for the design of humanoid robots. Robotics and Autonomous Systems, 37(2), 185-193.
  • Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., and Thelen, E. (2001). Autonomous mental development by robots and animals. Science, 291(5504), 599-600.
  • Zlatev, J., and Balkenius, C., 2001, Introduction: why epigenetic robotics? In C. Balkenius, J. Zlatev, H. Kozima, K. Dautenhahn and C. Breazeal (eds), Proceedings of the First International Workshop on Epigenetic Robotics, Vol. 85, Lund University Cognitive Studies, pp. 1-4.
  • Ugur, E., Dogar, M. R., Cakmak, M., and Sahin, E. (2007, July). Curiosity-driven learning of traversability affordance on a mobile robot. In Development and Learning, 2007. ICDL 2007. IEEE 6th International Conference on (pp. 13-18). IEEE.
  • Max Lungarella, Giorgio Metta, Rolf Pfeiffer and Giulio Sandini, Developmental robotics: a survey. Connection Science 15(4): 151-190, 2003.
  • M. Asada, K. Hosoda, Y. Kuniyoshi, H. Ishiguro, T. Inui, Y. Yoshikawa, M. Ogino, and C. Yoshida. Cognitive developmental robotics: a survey. IEEE Trans. Autonomous Mental Development 1(1): 12-34, 2009.
  • Chapters 1-4, Cangelosi, A. and Schlesinger, M. Developmental robotics: From babies to robots. MIT Press. 2015
  • Pierce, David, and Benjamin J. Kuipers. "Map learning with uninterpreted sensors and effectors." Artificial Intelligence 92.1 (1997): 169-227.
  • P-Y. Oudeyer, and F. Kaplan. "What is intrinsic motivation? a typology of computational approaches." Frontiers in neurorobotics 1 (2007).
  • P.-Y. Oudeyer, F. Kaplan, and V.V. Hafner. "Intrinsic motivation systems for autonomous mental development." Evolutionary Computation, IEEE Transactions on 11.2 (2007): 265-286.

  • L. Jamone, E, Ugur, A, Cangelosi, L. Fadiga, A. Bernardino, J. Piater and J. Santos-Victor, Affordances in psychology, neuroscience, and robotics: a survey, IEEE Transactions on Cognitive and Developmental Systems, under review.
  • E. Ugur, E. Oztop and E. Sahin, Goal emulation and planning in perceptual space using learned affordances, Robotics and Autonomous Systems, 59 (7-8), pp. 580-595, 2011.
  • E. Ugur and E. Sahin, Traversability: A case study for learning and perceiving affordances in robots, Adaptive Behavior, 18(3-4), pp. 258-284, 2010.

Reinforcement Learning:
  • Konidaris, George, Scott Kuindersma, Roderic A. Grupen, and Andrew G. Barto. "Autonomous Skill Acquisition on a Mobile Manipulator." In AAAI. 2011.
  • Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature 518.7540 (2015): 529-533.
  • Kober, Jens, J. Andrew Bagnell, and Jan Peters. "Reinforcement learning in robotics: A survey." The International Journal of Robotics Research (2013)

Deep learning
  • Lenz, Ian, Honglak Lee, and Ashutosh Saxena. "Deep learning for detecting robotic grasps." The International Journal of Robotics Research 34.4-5 (2015): 705-724.
  • Sigaud, Olivier, and Alain Droniou. "Towards Deep Developmental Learning.", IEEE Transactions on Cognitive and Developmental Systems (2016).

Policy search / Reinforcement Learning:
  • Kober, J., Oztop, E., & Peters, J. (2011, July). Reinforcement learning to adjust robot movements to new situations. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence (Vol. 22, No. 3, p. 2650).
  • Peter Pastor et al. "Skill learning and task outcome prediction for manipulation." Robotics and Automation (ICRA), 2011 IEEE International Conference on. IEEE, 2011.
Inverse Reinforcement Learning:
  • Ng, A. Y., Russell, S. J., et al. (2000). Algorithms for inverse reinforcement learning. In Icml, pages 663–670.
  • Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning (p. 1). ACM.

Learning from Demonstration:
  • Argall, Brenna D., et al. "A survey of robot learning from demonstration." Robotics and autonomous systems 57.5 (2009): 469-483.
  • Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009, May). Learning and generalization of motor skills by learning from demonstration. In Robotics and Automation, 2009. ICRA'09. IEEE International Conference on (pp. 763-768). IEEE.
  • Calinon, Sylvain, Florent Guenter, and Aude Billard. "On learning, representing, and generalizing a task in a humanoid robot." Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on 37.2 (2007): 286-298.

  • Miller, A. T., & Allen, P. K. (1999). Examples of 3D grasp quality computations. In Robotics and Automation, 1999. Proceedings. 1999 IEEE International Conference on (Vol. 2, pp. 1240-1246). IEEE.
  • Kao, I., Lynch, K., & Burdick, J. W. (2008). Contact modeling and manipulation. In Springer Handbook of Robotics (pp. 647-669). Springer Berlin Heidelberg.
  • Prattichizzo, D., & Trinkle, J. C. (2008). Grasping. In Springer handbook of robotics (pp. 671-700). Springer Berlin Heidelberg.
  • Roa, M. A., & Suárez, R. (2015). Grasp quality measures: review and performance. Autonomous robots, 38(1), 65-88.

Grasp learning:
  • Coelho Jr, J. A., & Grupen, R. (1997). A control basis for learning multifingered grasps
  • Detry, R., Ek, C. H., Madry, M., Piater, J., & Kragic, D. (2012, May). Generalizing grasps across partly similar objects. In Robotics and Automation (ICRA), 2012 IEEE International Conference on (pp. 3791-3797). IEEE.
  • Detry, R., Kraft, D., Kroemer, O., Bodenhagen, L., Peters, J., Krüger, N., & Piater, J. (2011). Learning grasp affordance densities. Paladyn, Journal of Behavioral Robotics, 2(1), 1-17.

Probabilistic Approaches:
  • Koppula, H. S., Gupta, R., & Saxena, A. (2013). Learning human activities and object affordances from rgb-d videos. The International Journal of Robotics Research, 32(8), 951-970.
  • T. Nakamura, T. Nagai, and N. Iwahashi. "Grounding of word meanings in multimodal concepts using LDA." In Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, pp. 3943-3948. IEEE, 2009.
  • Celikkanat, H., Orhan, G., & Kalkan, S. (2015). A probabilistic concept web on a humanoid robot. Autonomous Mental Development, IEEE Transactions on, 7(2), 92-106.

Human-in-the-loop learning:
  • Babič, Jan, Joshua G. Hale, and Erhan Oztop. "Human sensorimotor learning for humanoid robot skill synthesis." Adaptive Behavior (2011).
  • Peternel, L., Petrič, T., Oztop, E., & Babič, J. (2014). Teaching robots to cooperate with humans in dynamic manipulation tasks based on multi-modal human-in-the-loop approach. Autonomous robots, 36(1-2), 123-136.

Skill/symbol acquisition:
  • Mugan, Jonathan, and Benjamin Kuipers. "Autonomous learning of high-level states and actions in continuous environments." Autonomous Mental Development, IEEE Transactions on (2012): 70-86.
  • Konidaris, G., Kuindersma, S., Grupen, R. A., & Barto, A. G. (2011, April). Autonomous Skill Acquisition on a Mobile Manipulator. In AAAI.
  • E. Ugur and J. Piater, Bottom-Up Learning of Object Categories, Action Effects and Logical Rules: From Continuous Manipulative Exploration to Symbolic Planning, IEEE International Conference on Robotics and Automation (ICRA), Seattle, Washington, 2015, pp. 2627-2633