I recently watched the session, “Reality Check: Getting to Scale” presented by John Simmons, Head of Product, InOrbit at the Robot Operations Conference. The talk was focused on John’s experiences in Operations at Bossa Nova and the lessons they learned as they scaled the robot fleet from 5-350 units. Bossa Nova manufactures inventory control robots for use in retail stores.
Problems Evolve as You Scale
One of the points John made was that the type of problems you face changes during different phases of deployment.
When you field your first 5 robots, you are really just trying to get the system to some basic level of functionality in the target environment. You are probably still heavily in an R&D mode and are working to productize your prototype.
From 5-50 fielded systems, the robot is actually able to perform the job you designed it for. These systems are often deployed as part of pilots. The customers are likely early adopters of technology, understanding of the immaturity of the systems and willing to provide feedback on the product.
Once you get to the 50-350 range of deployed systems, you are likely out of the pilot phase. New customers are less forgiving of mistakes. They don’t want their job to be made more challenging by the introduction of your robot into their workflow. You may need to train users on how do to their jobs in the presence of the robot. At this phase, there’s also a higher likelihood of unexpected scenarios that may cause technical or operational issues the system design didn’t account for. At this scale, the focus is on proving that you have the tools and processes to scale the deployments and operate efficiently. The problems you identify are more likely to be process problems than deep technical problems.
The last phase John talked about was the 350-1,000 deployed systems phase. The goal here is to demonstrate the ability of the product to drive ROI for the customer. The customer is purchasing the robot services in order to achieve some business goal. At this scale of deployment and maturity of the product, you should be able to demonstrate the business impact the deployed systems are delivering.
I found this framework useful since most deployments will transition through these phases just maybe not at the exact same deployment sizes. Having an understanding of the focuses of each phase and the challenges associated with upcoming phases is helpful to ensure proper product roadmap development.
Maturing the Operations Organization
John mentioned that the robotics industry needs to become a highly stable, exception driven industry. He discussed how initially, Bossa Nova assigned individual human observers for each robot to monitor the robot onsite and support troubleshooting. Remote support was executed by an operations team that was technically sophisticated and relied on a complex set of developer tools to troubleshoot issues. These tools connected directly to the robot and were often slow and difficult to work with.
As the scale of deployments grew, these solutions were not scalable. They removed the human observers and were then exposed to a series of new issues that weren’t encountered previously when the observers were present. They began to distinguish responsibilities between initial support technician roles and the more technical technicians. Relying on lower skilled technicians to provide the initial tier of support lowered costs but also meant that they need to invest in simpler debugging and troubleshooting tools. The original tools were generally overkill most of the time so the simpler tools also improved response time. They eventually moved the initial onsite and remote support to a 3rd party, further reducing the support costs. In order to do this effectively, they needed reliable systems but also the training materials and mature product documentation to enable the 3rd party to be effective.
John also distinguished between two types of problems found in robotics deployments. In the beginning, there are a series of very technical problems that need to be solved by the engineering team. The logging and monitoring infrastructure may make it difficult to answer basic questions regarding the types of issues that are occurring and the root cause. As the deployments scale, the Operations team needs to provide more actionable data to engineering. They need tools and processes that can aggregate data from across the fleet and provide data-based list of priorities to the engineering team. This includes getting reliable data to show what was escalated to engineering and better metrics around resolution times. In addition to more detailed data, the Engineering and Operations team will need to collaborate more closely to reduce the time to resolve issues.
As the more frequent technical issues are identified and addressed, the scale of the deployment exposes other issues which are categorized as operational problems. These can be addressed with operations management tools. This can include defining processes, runbooks, or SOPs so that lessons learned from the field can be captured and the overall efficiency of operations support can be improved.
I enjoyed this talk and the perspective John provided on deploying robotics systems at scale. As more and more robotics systems are deployed in the real world, increasingly in semi or unstructured environments and in collaboration with people, it’s important to share best practices so we can avoid common pitfalls and speed up the adoption of these systems.