Brown CS 2951-P

Brown CS 2951-P

Human-Robot Interaction Seminar

Fall 2014

Building Human-usable 3D Maps for Robots


CS 2951-P is a seminar course on Human-Robot Interaction (HRI). The theme of the HRI Seminar in Fall 2014 is "Building Human-usable 3D Maps for Robots". Consider how interactive 2D maps of the world have become commonplace, almost essential, in our daily lives. Map services from companies such as Mapquest, Google, Apple, etc. are used in many different ways to help our decision making about where to go, where to shop, where to live, and a plethora of our activities. Maps from these services contain not only physical descriptions of space (such as geospatial locations, metric geographic terrain, street views) but also important semantic information (navigation graphs, directory labels, ratings). Such maps are also the basis for autonomous decision making for advising people as well as for robot control. For example, route directions provided by a mapping service between locations is generated by a planner that searches for optimal paths in the map. Similarly, a robot will use a planner on a map to autonomously move across the locations in a building. However, this requires a map building phase followed by a labeling phase and then a planning phase. Building on advances in depth sensing technologies, we will focus on building maps using Red-Green-Blue-Depth (RGBD) sensors, such as the Microsoft Kinect, followed by interactively labeling these maps so that they can be used to plan and execute a task by robots such as the Willow Garage PR2.

This course focuses on SLAM part of robotics and how human interaction on these maps would help in building better maps. We will cover topics that address the perceptual challenges involved in robotics for estimating 3D maps. The course will go over basic concepts in robot sensing (from odometry, laser rangefinding, RGBD sensors and cameras) and robot mapping and state estimation (including Simultaneous Localization and Mapping, or SLAM). We will additionally cover new exploratory ideas for semantic labeling of maps through recent publications in robot supervisory control, robot learning from demonstration, semantic mapping, geometry sketching, etc. We will essentially discuss research papers covering the following areas:

Course Staff


Prof. Chad Jenkins
CIT 327
Office Hours: M 1:30-2:30pm, Tu 1:30-3pm

Teaching Assistant

Karthik Desingh
CIT 315
Office Hours: Fr 10am-12pm


Prof. Jenkins administrative support person is Suzanne Alden.

Meeting time/place

Monday 3:00-5:20 (M Hour)
CIT Lubrano (Class session) and CIT 115 (Lab session)

Mailing list:

Blogpost discussion:

Optional Textbooks

Multi-view Geometry in Computer Vision, Hartley and Zisserman, Cambridge University Press, 2004.

Telerobotics, automation, and human supervisory control, Sheridan, MIT Press, 1992.


Linear Algebra (CS 53 or equivalent)

and one of:
Autonomous Robotics (CS 148), Computer Vision (CS 143), Computer Graphics (CS 123), Artificial Intelligence (CS 141)

or instructor permission

Schedule (tentative)

Class session
Lab session
1 Sept 8 Initialization
Introduction, Paper assignments, etc.
Instructor RGBD SLAM demos Slides Prelim Projects 1, 2, 3 - Out
2 Sept 15 Prelim Project Primer

In-class refresher: 3D transforms and odometry

Fast Iterative Alignment of Pose Graphs with Poor Initial Estimates, Olson et al., International Conference on Robotics and Automation, 2006.

Point Set Surfaces, Alexa et al., IEEE Visualization, 2001.


Introduction to Point Cloud Library
Refresher Slides


3 Sept 22 Hacking session on Prelim Projects Hacking session on Prelim Projects
4 Sept 29 ICP-based Mapping

RGBD Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments, Henry et al., International Symposium on Experimental Robotics, 2010.

3D Mapping with an RGB-D Camera Endres et al., IEEE Transactions on Robotics, 30(1), 2014.

Hacking session on Prelim Projects RGBD-Mapping_Henry

3D Mapping_Endres
5 Oct 6 Keyframe-based Mapping

On unifying key-frame and voxel-based dense visual SLAM at large scales, Meilland and Comport, IEEE Intelligent Robots and Systems, 2013.

LSD-SLAM: Large-Scale Direct Monocular SLAM, Engel et al., European Conference on Computer Vision (ECCV), 2014.
Jun Ki Lee

Hacking session on Prelim Projects Unifying Key-frame and Voxel based SLAM

6 Oct 13 Fall Weekend Holiday! Prelim Projects due; Project 4 - SLAM out
7 Oct 20 Mapping by Factor Graph Optimization

iSAM: Incremental Smoothing and Mapping, Kaess et al., IEEE Transactions on Robotics, 24(6) 2008.

Inference on networks of mixtures for robust robot mapping, Olson and Agarwal, Robotics: Science and Systems, 2012.

Hacking session on SLAM project iSAM

8 Oct 27 3D Reconstruction

KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera Izadi et al, ACM UIST, 2011.

PTAM, George Klein et al., ISMAR, 2007.

Hacking session on SLAM project KinectFusion

9 Nov 3 Geometry by Annotation

3-Sweep: Extracting Editable Objects from a Single Photo, Chen et al., ACM SIGGRAPH Asia, 2013.

VideoTrace: Rapid interactive scene modelling from video, van den Hengel et al., ACM SIGGRAPH, 2007.

KML - Keyhole Markup Language tutorial and hands-on exercise.

Hacking session on SLAM project VideoTrace
10 Nov 10 Filter based SLAM

FastSLAM, Michael Montemerlo et al, AAAI 2002.
Grasping and Affordances

Strategies for Human-in-the-Loop Robotic Grasping, Hsiao et., HRI, 2012.

Localizing Handle-like Grasp Affordances in 3D Point Clouds, Platt et al., ISER 2014

NASA Valkyrie Affordance Templates. Paper draft available as hardcopy through course staff.
Joey - FastSLAM FastSLAM
11 Nov 17 Hacking session on final project Hacking session on final project
12 Nov 24 Thanksgiving Break! Hacking session on final project
13 Dec 1 Strategies for Human-in-the-Loop Robotic Grasping, Hsiao et., HRI, 2012. Joey Hacking session on final project Strategies in Grasping
14 Dec 8 Final Project demos and Presentation Final Project due.

Projects and Grading

Full participation in this course consists of presenting 2 papers in class, participating in paper discussions, and completing 5 projects. The 5 projects are categorized as:

CS 2951-P projects are graded as “checked” (completed) or “not checked” (incomplete). Please create a repository for your work (preferably on github) such that it can be retrieved by the course staff. All the project deadlines will have a demo by the students in the lab session. Final project will also require a presentation regarding the various methods used and results. The final project involves the design and implementation of a human-usable interface for labeling a map built from the course programming assignments.

Paper discussions and submission of paper summaries will occur through the course discussion forum. For full credit, the summary of a given paper must be entered into the discussion forum before the presentation of this paper in class.

To receive an A grade in the course, you must:

To receive an B grade in the course, you must:

Git repositories will be used for project implementation, version control, and submission. Project implementations are submitted as branches in your assigned repository, named "projectX" for X project. These branches must be submitted prior to the due date for each assignment. Your implementation will be checked out and executed by the course staff. You will be notified by the course staff whether your implementation is sufficient for checking off the assignment. If your assignment is insufficient for receiving a check, your group is allowed one regrade (per assignment) with 2 weeks of notification. If deemed necessary, the course staff may require an interactive demonstration of your implementation and/or a web-based written report.

Final Grading

All grading will be finalized on December 10, 2014, after final project presentations.


The course staff will provide a git repository (through github) for each student group to check in project source code and reports, unless you request using separate service or repository. Please refer to this tutorial for an in-depth introduction to git and version control. Although it has a coarse language choice, this is also a great and accessible tutorial. We expect students to use these repositories for collaborative development as well as project submission. It is the responsibility of each student group to ensure their repository adheres to the Collaboration Policy and submission standards for each assignment. Submission standards and examples will be described for each assignment.

Late Policy

Do not be late submitting assignments. The course staff reserves the right to not grade late submissions.

Collaboration Policy

This policy covers all course material and assignments unless otherwise stated. Course material information and documentation may be discussed with anyone. Course material does not include assignment handouts. Assignments may be discussed with the other CS 2951-P students. Discussions may make use of a whiteboard or paper. Discussions outside of your group cannot include writing or debugging code on a computer. You may take notes away from these discussions, provided these notes do not include any source code. The code for your implementation may not be shown to anyone outside of your group, including granting access to repositories or careless lack of protection. You do not need to hide the screen from anyone, but you should not attempt to show anyone your code. When you are done using any robot device such that another group may use it, you must remove all code you have put onto the device. You may not share your code with others outside of your group. At any time, you may show others the implemented program running on a device or simulator, but you may not discuss specific debugging details about your code while doing so. Should you fail to abide by this policy, you will receive no credit for this course. Brown also reserves the right to pursue any means necessary to ensure compliance. This includes, but is not limited to prosecution through Brown University’s Student Conduct Procedures, which can result in your suspension or expulsion from the University