While obstacle detection and avoidance have largely been solved for mobile robots, perception and manipulation remain challenging for piece-picking operations. Advances in computer vision, machine learning, and end-of-arm tooling are now enabling robots to pick at human and even superhuman rates.
One company focusing on successful automation of pick-and-place workflows is Material Handling Systems Inc. (MHS). The Mt. Washington, Ky.-based company spun out MHS Robotics from its research and development unit last year.
Kristiyan Georgiev, R&D manager for computer vision and software engineering at MHS, spoke with Robotics 24/7 about improvements in robotic picking applications.
How long have you been at MHS, and how does your prior experience help you manage these technologies?
Georgiev: I joined MHS three years ago. My career has ranged from medical devices to high-frequency trading, and I've been at companies big and small, including startups.
The common theme among all these positions is tackling complex problems in difficult situations. The main difficulty in singulation of parcels using robots is that not every parcel is the same. There's no model that you can build to provide accuracy for all of them.
Robots must be able to handle items without having seen them before, operating on the fly. There are also corner cases, like a loose wire or a signal not getting to the sensor. You have to be able to question the premise of a new device and difficulties around it.
As for execution, I got my Ph.D. in autonomous ground vehicles and SLAM [simultaneous localization and mapping]. I focused on these features in my dissertation, then on things involving 2D vision and AI.
How much growth in demand has MHS seen from e-commerce acceleration in the past two years?
Georgiev: Quite a bit—we're experiencing 40% to 50% growth every year, and it was an even bigger number during the pandemic.
Which sectors can benefit most from newer automation—consumer packaged goods, pharmaceuticals, or something else?
Georgiev: Three core categories can benefit from these systems. The first is the cost of labor. Is the technology reducing the amount of labor required to operate?
Distribution and fulfillment centers are good candidates for automation because they're labor-intensive. As Amazon has been growing, the number of its employees has grown alongside that of robots because of the growth in online ordering.
The second is where there's a lack of qualified workers, such as in skilled trades for manufacturing. Robots are good candidates for repetitive tasks, such as welding, assembly, or machine tending. A CNC machine takes a block of metal and removes piece of it to achieve the desired shape. A robot can then take it out and put in a new block.
The third is where product quality or process efficiency matter. Pharmacies are good candidates for automation because it's not efficient for customers to wait to pick up their prescriptions. We'd love to have a machine do it. I've heard horror stories of incorrect medicines because of human error, where the customer didn't find out until afterwards.
How have MHS's picking systems improved over the past few years?
Georgiev: We've improved quite a few things in our robot singulator. Throughput has nearly doubled. We use the same exact robot and layout, but smarter software makes better decisions over time.
Another area is mis-sorts. The machine takes one package from a pile of packages and inducts it into a sorter. A single package has to be small enough to fit and oriented the right way.
Previously, we struggled with doing it 100% of the time and had errors. For example, the robot might take two packages, which is a problem for sorting. We reduced the number of errors by 10 times.
MHS's algorithm is now able to handle a wider variety of packages—including some not in specifications or what's expected. This includes large parcels and different textures, materials, shapes, prints, etc.
Speaking of edge cases or exceptions, how much are people still in the loop for robotic picking?
Georgiev: The amount of time between required interventions by humans are most important. Robots are not here to replace jobs but to complement workers. Humans are a lot more agile and can adapt, react, and think outside the box.
If something not up to spec occurs—such as a large box or a torn one—humans can see it and take it out. Cases where you need a person include when there's trash on the conveyor belt or missing labels.
The rest of the time, you want robots to do the repetitive task of picking and placing individual packages. To reduce interventions, we've built better controls and communications for the robot to interface to other machines and the facility.
Which technology has changed the most—machine vision, machine learning, gripper technology, or motion control?
Georgiev: We've improved the software and can replace the physical robot arm, and they still perform the same tasks. We've deployed this at a few locations already
By changing the layout and recognizing where there's a bottleneck—the camera confirms the pick of a single package. While there are physical limitations of speed, acceleration, and the robots' capacity, with two robot arms, we can increase the throughput.
We have deployments in four locations, including in one small contract and one large contract, with one to five very large contracts expected this year.
The vision software MHS has built in house has improved the most. We've transitioned from a hybrid approach relying more on neural networks to understand the scene and types of packages and make better decisions on how to pick them and how to orient end effectors.
Certain suction cups have better properties for picking loose polybags, and the system knows where to place it and the overall order of picking.
For the multi-pick end effector, what size or sorts of items are most commonly fulfilled? How does its throughput compare with current industry norms?
Georgiev: Our end effector ranges from small envelopes of 3x3 in. flat to bigger boxes of 20x20x20 in.
We tell the robot where to pick and how to pick. It can dynamically configure the motion of the robot, depending on the object.
We've observed different mail mixes. Some days, it's mostly polybags; others, mostly boxes. MHS tends to do better with boxes, which have a defined shape. Polybags might be perceived as oversized based on their shape—a human might just fold a polybag, but a robot might reject it.
We now have three years of data, comparing one robot shift with a shift of human operators. Humans are more dexterous and can handle short intervals, but intense activity is not sustainable. In the long run, robots do slightly better than people.
Can you briefly describe how the robotic singulation and induction system retries in the case of unsuccessful picks and handles exceptions?
Georgiev: If we fail, we reattempt up to three times. It depends on if the system cannot establish contact with the package, or if we were able to grasp it but it's too heavy.
We have a patented reject mechanism of a paddle on the belt. It handles the majority of exceptions, such as if an item is too big or too high, or it's too porous for suction cups.
If all of these fail, we call the local operator to resolve the issue.
Does MHS offer its systems for sale or through a robotics-as-a-service (RaaS) model?
Georgiev: While RaaS is popular, we sell systems as capex [capital expenditures]. We're in traditional automation and sell machines directly. It's better for ROI [return on investment] because the device monitors itself.
What are some examples of applications or types of orders that are best addressed with dual-robot each picking from totes?
Georgiev: We've only scratched the surface for applications. For the past four years, MHS has focused on parcels for express couriers.
With dynamic neural networks, it's now more possible to pick from random totes for fulfillment.
How much of the training on parcels or SKUs is done by MHS, an integrator, or the end user?
Georgiev: We work with outside contractors to supplement any limitations on resources. For this project, we've done it 100% internally, with millions of images collected from actual sites.
They're annotated and used to train the networks. We continually collect and annotate data to train and reduce errors.
The types of images we're looking at are both 2D and 3D. We developed our own industrial-grade camera five years ago.
How much do MHS's sortation systems communicate with inventory or packaging software?
Georgiev: Our robot singulation system communicates with only the sorter in the facility. The singulator inducts each package.
If we take it out from the express courier where the robots are and apply to the DNF [software package manager], then there's a need for integration with inventory software.
Currently, we're only building for the reservation of certain cells on the sorter. Beyond that, what happens to a package is beyond our control. We do keep stats on package dimensions for QA [quality assurance].
What are some things that you're working on for this year?
Georgiev: Most of the efforts we're doing right now is in decreasing the amount of human intervention.
We're looking at possible causes of corner cases—packages getting stuck or being too large for the machine to handle and needing to get off the belt.
Sometimes, the robot may collide with package. If that happens, it stops and asks for an operator to inspect it.
As companies focus on singulation, if they add robots to replace labor, but then add labor to attend to robots, the ROI calculation is imprecise.
What are your goals for interventions and throughput?
Georgiev: Since robotics is a relatively new technology, there is not really an industry-accepted metric for human interventions, but we try to look through eyes of customers.
For a facility with a single robot, if someone has to tend to it every 30 minutes, that's 50%, so it should definitely be less than an hour. We're actually aiming for one intervention a day.
The industry takes widely different approaches. Some companies are doing remote interventions, but sometimes you can't fix a problem remotely.
The high-level goal for our systems is to increase the throughput our system could handle. This involves not just improvements in the layout—for which MHS has plenty of experience as an integrator—but it also includes improvements in picking strategies. Sometimes robots are “starved,” and there are not enough packages in lane delivery. Or there are bottlenecks, which we can also solve.
We've solved about 95% of picking problems, and it will take quite a bit of effort to move from 95% to 96%. We've already addressed the “low-hanging fruit” of integration.
Related Resource
E-commerce Fulfillment Technologies
What technologies should you consider for for warehouse automation? Download Now!
About the Author
Follow Robotics 24/7 on Linkedin
Article topics
Email Sign Up
Related Picking News
Related Companies
Related Resources
