Modern product management experience is highly influenced by and optimized for purely software based products. While core principals translate well to products involving a combination of hardware and software, there are some unique considerations to be aware of when managing robotics products.
From Atoms to Bits
The social, mobile and cloud revolutions were some of the dominant technological shifts of the last two decades. For enterprise companies, this enabled the development of Software as a Service web applications. One the consumer side, the internet drastically reduced the costs of developing and distributing digital goods. With a limitless supply of digital goods, companies that aggregated these items and enabled the user to quickly discover what they were looking for were successful. Modern product management leverages the flexibility of software development to enable quick experimentation, allowing the product team to rapidly test and validate ideas to deliver more value to the end user.
Just because a product operates in the digital world doesn’t mean it can’t impact the physical world. Uber’s algorithms influence the movement of cars around a city as they balance the supply of drivers with the demand of riders. Amazon’s software identifies the optimal fulfillment center for your order and coordinates an entire supply chain to get the order delivered to your door. Some may argue that the robots are already here, they are just packed in data centers for heat and efficiency reasons. People are deployed at the edges of these digital networks, taking direction from the algorithms and handling what the computers are not yet capable of. Robotic applications will extend the impact of these digital systems into the physical world.
Artificial Intelligence is one of the next major shifts in computing. Humans are increasingly partnering with computers/AI to accomplish tasks. The internet resulted in a flood of text and other media which is now being used to train ML models for content generation (code: Github co-pilot, text: ChatGPT, images: Stable Diffusion). This partnership will inevitably transition into the physical world as well, as humans and robots cooperate collectively. Everyday Robots has a vision where helper robots are as useful in our physical lives as computers have been to our digital lives. (https://everydayrobots.com/thinking/rt-1-a-new-leap-in-robot-learning-at-scale)
Robotics will be a major growth areas as a beneficiary of these technology trends. More work is needed to enable robots to become useful in the messy and unstructured environments where humans operate. This post describes some considerations to be aware of that I’ve encountered developing robotics products. Most of these apply to other industries as well, but I think the combination is unique in robotics applications. These include:
- Hardware Lifecycle Management
- Software 2.0 and the Data Engine
- Product Feedback Challenges
- Assessing Product Safety
- Navigating Enterprise Sales
Hardware Lifecycle Management
Dealing with hardware isn’t unique to robotics products. However, for Product Managers used to purely software based products, there are some considerations to be aware of.
One of the most obvious differences is the slower pace of development and iteration on hardware components compared to software systems. Hardware components require manufacturing which is typically longer than deploying a software component. This has several implications for product roadmap development.
First, your roadmap planning will likely need to focus on a longer time horizon than if you were dealing with a purely software based product. While you can deliver software features independent of hardware changes, there is still a coupling between the two domains that you need to account for. You may be limited in which opportunities you can purse without changes to your hardware. Features may depend on more compute resources on the robot, higher bandwidth connectivity, or additional sensing modalities on the platform.
You’ll need to account for these changes, which could take years from initial planning to full production release, when prioritizing which product opportunities to prioritize. You’ll also need to devote software engineering resources to determine the requirements for this future hardware ahead of time as well as support the test and evaluation of these hardware components as they go through the various validation gates before they are integrated into the final product.
Recently, there’s been a shift away from time-based, gantt chart style roadmaps to time horizon roadmaps that list opportunities to address, not specific solutions or features. However, there’s still a need to communicate high integrity time based commitments which enable other departments and customers to align their schedules appropriately.
There are a large number of long lead time items that support a major hardware revision. The supply chain team needs to order components for assembly. Manufacturing may need to adjust tooling to build and QA the component. Safety and reliability tests get scheduled assuming prototype samples will be available by a certain date. Delaying a release can waste a lot of time and money internally on things like excess inventory or expedite fees.
On the Engineering side, you’ll have to determine how much backwards compatibility to maintain for new software operating on the soon to be legacy hardware. Providing maintenance support for two generations of hardware is taxing. You’ll start by spending resources bringing up the new hardware so that it meets performance parity of the legacy system, which distracts focus from supporting the currently fielded system. Once the new system is released, you’ll then need to make tough decisions about how much effort to put into issues that only impact the legacy system. You’ll need to make sure your software test and release process passes test suites on both versions of the hardware. At some point, you will end of life the legacy system and you may need to determine what to do with any left over legacy systems.
There are also go to market implications associated with hardware releases. When you announce the new hardware, customers may opt to wait for the new hardware instead of purchasing more of the soon to be legacy system. Significant delays in the release of the new system could mean you sit on a large inventory of legacy product that you now may need to discount to sell, lowering your revenue projections. On the other hand, some customers may not want to upgrade and want continued availability and support for the legacy system. You’ll also need to make sure you clearly communicate which functionality will be deprecated on the new system.
Externally, customers may require significant changes to their system, including re-validating or certifying their system, to accommodate your new product. Delaying a release can waste a lot of time and money internally on things like excess inventory or expedite fees and can cause frustration with your customers. You’ll need to be comfortable committing to release dates and functionality further out than you are typically used to, and driving execution to ensure the status of those commitments are regularly and accurately communicated.
Software 2.0 and the Data Engine
Robotics applications are transitioning from the highly constrained and controlled environments like manufacturing facilities into more open and unpredictable environments like sidewalks or public roads. As robots interact more in the “real world”, the problems presented have the property that it is significantly easier to collect the data than to explicitly write the program to get the desired robot behavior.
As a result, more of the robotics stack software development is shifting from the explicit definition of algorithms in C++ to the tuning of weights in neural networks for functions like perception, prediction, and planning. Engineers don’t code the weights directly. Instead they select the model architecture, modify the loss functions, and manipulate the dataset to achieve the desired output of the system.
Tesla refers to the feedback loop for their ML algorithms as the “data engine.” They receive a continual stream of data from their fleet of vehicles and user behavior to identify scenarios where the vehicle doesn’t operate as expected. They then mine their datasets to identify a large number of examples of the scenario of interest. They rely on simulation, human labelers, and automated labeling processes to generate test and evaluation datasets and re-train their models. They then deploy this update to the fleet and monitor performance.
If your robotic application relies on ML algorithms or you plan to rely on ML algorithms in the future, investing in your own data engine is critical. If you decide to build these tools and workflows in house, the platform product manager role will be vital to making the end to end feedback cycle functional and scalable. You’ll need to develop cost efficient mechanisms for getting the appropriate data (potentially from high bandwidth sensors) off the deployed system. You will need tooling to support aggregating and visualizing the data. You will have to consider which data annotation tasks are best suited for humans and which can be processed by offline algorithms. There will be a constant effort to increase the quality of the dataset, making sure it’s large, diverse, and clean. These tools and workflows aren’t directly user facing, but they need to be supported with as much priority as aspects of the product with direct user interaction.
Product Feedback Challenges
It’s not just the ML algorithms that require a continual stream of data and feedback. Product teams also want data from the field to understand user behavior to improve the product and deliver business results. Robotics platforms pose some unique challenges to executing a tight build, measure, learn feedback loop that is desired to quickly iterate on a product.
Robotics platforms sense their environment in order to interact with it. As they operate in more unstructured environments, they may require a large quantity and variety of sensors. These sensors can produce large volumes of raw data. It may be expensive to offload, process, and store all of this data. If, for example, a user encountered an error when interacting with your product, you may want all the raw data leading up to the error in order to diagnose the issue.
Investing in a triggering system that can identify subsets of the data to be offloaded can help at the risk of producing false negatives and losing data you want. Even if you get all the data you need, you may require offline systems or humans to annotate the data in order to analyze it properly. If you are evaluating your system’s performance on a relatively rare scenario, it may take a while to even get enough data back to complete your analysis and make a decision. All of these steps introduce latency and opportunities for error when collecting data. You can invest in training more people to manually process the data to reduce latency. It’s often times easier to train someone to review the data and tag or segment portions that align with the attributes you are interested in. Ideally you can develop algorithms to replicate the human processing to reduce the latency and the variability on annotation quality. Tight coordination between the Operations and Engineering teams are key, as the Operations team can pass along a lot of context with the issues they observe and report.
Assessing Product Safety
Historically, robot systems have been optimized for a limited set of tasks and deployed in controlled environments like manufacturing facilities. As robotics platforms mature in their functionality, they tend to interact more directly with their users or operate in more unstructured environments around people. This increases the burden on the robotic system developer to ensure their system doesn’t put the safety of the users or other people in the environment at risk. You may be able to build an initial product without the appropriate focus on safety, but it will be challenging to build a business without it. This challenge is similar to other highly regulated industries like healthcare or finance. However, for some robotics applications, the safety frameworks are still evolving to account for products relying on ML in highly unstructured environments.
Safety engineering dictates that we define a risk goal that defines how much risk we are willing to accept and what is safe enough. For conventional systems where it’s possible to fully characterize the Operating Domain and enumerate the known risks, this is more straightforward. You can identify all the risks, mitigate them to an acceptable level, and deploy your system with little uncertainty.
This breaks down when our robotic applications operate in open and unstructured environments, relying on ML models for safety critical capabilities. We still need to set a risk goal and follow industry standards to mitigate the residual risk to an acceptable level. But we need to address the uncertainty in our risk estimate that can arise from gaps in our testing, unexpected edge cases, biases in our ML model training data, or unexpected ML model behavior.
One approach is to operate the system in controlled test environments and monitor the rate of unexpected failures. We can address the failures until we stop seeing failures. This assumes we can discover enough of the initial unknowns to meet the safety goal and deploy. We can attempt to monitor the uncertainty after we deploy by tracking a set of safety related leading indicators that can signal to us that our assumptions might be valid or a risk isn’t properly mitigated. Our system won’t be perfect when we deploy, but we want it to be safe enough to be ethically responsible to deploy.
From a product roadmap perspective, you need to account for time to test and validate new features before they are available to your users. Your validation process may be time consuming and expensive, which may result in a slower release cadence. You’ll likely need to devote internal resources to the development of a robust suite of testing tools, like simulation, to ensure you can comprehensively test your application in a variety of scenarios.
It’s a challenge to balance engineering resources between feature development, and supporting the safety analysis and testing initiatives, especially if you don’t have a fully featured product yet. You’ll need to recognize which subsystems are stable enough to invest in formal documentation, analysis, and test development.
Navigating Enterprise Sales
While there are consumer robotics success stories like iRobot, the majority of robotics companies today are business to business focused. There are several lessons learned selling to enterprise markets that apply just as well to robotics products as they do for traditional software products.
One thing to understand is that being a new technology might be great for a consumer market, but business customers view something new as a risk. Large companies have likely developed their own internal tools and processes. It will cost them time and money to adjust to your new technology. Therefore, you need to spend your time and money developing a product that can easily integrate with their current system. The result is that your solution is either overly optimized for a single customer application making it difficult to extend to new customers or highly flexible and configurable which may make it complex and less easy to use.
These unique integration requirements pose a challenge to product roadmap development. The Product team may be pressured to commit to specific features that were unplanned or deprioritized in order to close a deal. We need to be thoughtful in our responses and what we commit to. Due to the nature of enterprise sales, we’re going to be working with this specific customer for a long time. Once we make a commitment, they will likely force us to meet that commitment. Each promise to a customer creates chaos and churn. These requests can quickly consume a non trivial amount of time and staffing, putting our planned roadmap at risk. Matters can be made worse if we are asked to commit under time pressure and aren’t able to complete the proper discovery efforts. Executing on these raw requirements may not even solve the original problem while taking time away from initiatives on the roadmap that were properly reviewed and prioritized based on our strategic objectives.
From a user perspective, the person interacting with your product on a daily basis is not likely the person making the purchasing decision. Therefore, you need to demonstrate value to the direct user of your product as well as make a business case to the group or individuals who own the budget. These companies may have made large capital investments in their current systems. Their may also be significant internal political investment in the decisions associated with their current system which resist your solution. You’ll need to know whose budget the purchase will come from and who has veto power over the deal. Who’s reputation is at risk for choosing the wrong product? Who are the internal champions for your competitors solutions?