HERB, the Home-Exploring Robot Butler, uses color video, a Kinect depth-sensing camera, metadata and other non-visual information to understand objects.Carnegie Mellon University Robotics Institute
Future service robots will need to quickly discover objects in their environment to maximize their potential. But robots that rely on computer vision alone often struggle to fully define objects in their surroundings.
Therefore, a research team from Carnegie Mellon University’s Robotics Institute has developed Lifelong Robotic Object Discovery (LROD), a process that enabled HERB, the two-armed Home-Exploring Robot Butler, to use color video, a Kinect depth-sensing camera, metadata and other non-visual information to more accurately refine its understanding of objects. The process ultimately paves the way for a more autonomous robot, capable of identifying its environment and the objects in it on its own.
“We want to make a robot we can send to somebody’s home and that robot would be able to start doing meaningful tasks in that home without training,” Alvaro Collet, a robotics doctoral student at CMU and one of the lead researchers, told Discovery News.
Metadata, or "domain knowledge," such as location, timestamps, size, shapes, colors and whether objects could be lifted was gleaned from HERB’s video feed, which refined the robot’s algorithms and helped it identify 121 objects in a home-like laboratory. Not only did the process nearly triple the number of objects HERB could identify, it decreased processing time by 190 times.
Traditionally, roboticists must build complex digital models and images of objects, then manually load them into the memory of robots. Researchers say this process is time consuming and not feasible for potential users of service robots, for example the elderly. With the implementation of LROD, the robot’s “HerbDisc” is expected to gradually refine its own models in order to better assist people on a daily basis.
Object recognition has been challenging for roboticists and computer vision researches because cluttered environments can easily muddle robot computations that are based solely on visual data. While shapes can be determined, the intended functions and location of objects are harder to identify. Humans, for example, don’t rely on sight alone to understand objects. We pick them up, feel them and register when and where we use them to create our own ‘domain knowledge.’ Therefore, we understand that a frying pan is often found and used in the kitchen, not in the bedroom.
The same idea is applied to HERB’s domain knowledge, made possible by LROD’s complex system of algorithms to process metadata. Also included in HERB’s domain knowledge is an object’s placement location, whether it’s on a table, the floor or in a cupboard. By using its arms, HERB can verify if an object can be moved or lifted -- the ultimate definition of its “objectness.”
“The way that object discovery algorithms work is that you essentially look into everything you see,” said Collet. “In the case of our robot, it’s a video stream of image data. You look at all the images and try to find objects or entities that are similar throughout multiple images, so that when you see a cup that is green and has a logo on it, that cup will always be detected in the images because it has a defined boundary and location.”
Derek Hoiem, a computer science assistant professor at the University of Illinois, compares the challenge of object recognition to human development – how it takes years for infants to mature into fully functioning adults. Though he says we should expect the same of robots that act like humans, he believes LROD's HerbDisc is essential to speeding up the process.
“Enabling machines to gradually learn about the world through their own interactions is a key for creating machines that perform human-level tasks,” Hoiem explained. “This project brings us one step closer to having robot helpmates that will one day seem indispensable.”
Researchers envision HERB and other service robots could one day use the Internet to create a richer understanding of objects and their surroundings. Video of HERB's LROD system can be seen here.