Learning Geometry from Vision for Robotic Manipulation

• Author / Creator

  Jin, Jun

• This thesis studies how to enable a real-world robot to efficiently learn a new task by watching human demonstration videos. We propose to introduce a geometric task structure as an interpretable inductive bias to the learning problem. We aim to learn a representation that geometrically encodes “what the task is” from offline human demonstration videos and then transfer the learned representation to a robot controller using uncalibrated visual servoing (UVS). Specifically, we propose Visual Geometric Skill Imitation Learning (VGS-IL), which uses a graph-structured task function to learn a task representation under structural constraints in the form of a predefined graph prior related to a geometric constraint type. The task function is optimized by Incremental Maximum Entropy Inverse Reinforcement Learning (InMaxEnt-IRL) based on “temporal-frame-orders” in human demonstration videos.

  We show that the learned representation selects task-relevant image features to compose projective invariant geometric constraints, thus forming an efficient and interpretable representation. Secondly, the learned representation selects out the equivalent geometric constraints in the robot scene with adjoint geometric errors used in visual servoing controllers, thus removing the need for extra robot training when mapping the task representation to robot actions. Lastly, by building task specification correspondence, we show that the learned task function selects task-relevant geometric constraints on categorical objects with the same task functionality, thus achieving task generalization. This is proposed as Categorical Object Generalizable VGS-IL (CoVGS-IL). Various real-world experiments were conducted to verify our proposed method’s ability regarding sample efficiency and task generalization.

• Subjects / Keywords

  • robotic task specification
  • representation learning
  • task generalization
  • geometric task structures
  • learning by watching
  • visual imitation learning

• Graduation date

  Fall 2021

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/r3-kekg-0819

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.