Giving Spot computer vision goggles

In this post, we will run through how we can add various computer vision logic to a Spot “action”. A Spot action is a task that can be performed during teleoperation or programmed into a routine Autowalk Mission. The task will typically involve taking in an image captured by Spot or collecting other types of data. In this case, we’re more focused on taking a single photo and running computer vision on the image to extract data or run inference.

I would like to point you in the direction of the Boston Dynamics Spot SDK Conceptual Documentation as well as their GitHub Repository which contains plenty of examples you can run through. This post will reference some code snippets from these resources, however, they will be the best place for up-to-date code examples.

Network Compute Bridge Service

“The network compute bridge allows you to off-load computation to another computer on the network. For example, you might want to offload a deep neural network call to a server in the cloud.”

On top of this high-level concept, the Network Compute Bridge or NCB Service allows us to easily configure a Spot action that takes in an image source and allows us to run our custom logic on the requested image. We can then modify our output image and display it on Spot tablet. Of course, this also means we can save this image for future reference.

Using this functionality, we can add simple (or complex) computer vision logic to infer or extract data from the images we capture with Spot. 

An example of how this can be used to benefit your business can be found in our recent blog post Extending Spot's Preception with Computer Vision.

Service Customization

What if we need user input before or at the runtime of our computer vision work? In comes the service customization, with which we can add custom fields into our Spot action for the user to interact with. We can see this being used in the Thermal Inspection action by Boston Dynamics to define minimum and maximum temperature thresholds per region of interest. This allows us to build out the logic to allow operators to set and be alerted if our computer vision extraction reads values outside of the desired, i.e. a tank or reservoir minimum level or an analogue gauge min and max value.

Result

Once the Network Compute Bridge action has been created and deployed to Spot, during a routine inspection we can leverage the computer vision to extract data from what Spot sees.

In this example, we ask a user to set an expected color for the liquid within the tank and a minimum value. If either of these criteria is not met, we’re left with a “failed” inspection result which can be used to alert an operator.

This is an oversimplification of a possible real-world example, however, the concept remains the same.

Example code

Because the code to implement computer vision functionality into a Spot action, and the action code itself is a little long, we’ve opted to not simply do a code dump. For a detailed example, you can check out the Spot SDK’s Service Customization Network Compute Bridge example, or the NCB example without the service customization portion.

In conclusion

Boston Dynamics has created a very versatile dynamic sensing platform that can bring instant return on investment. However, much more value can be obtained by using machine learning techniques such as computer vision to extract data for business insights and predictive maintenance.

Osedea can also offer proof of concepts with Spot to see if it’s the right tool for you and your business. Send us a message, we’d love to discuss how we can collaborate.