WORLD – Microsoft and Liebherr have revealed details of their current collaboration as part of the Liebherr household appliances division’s digital initiative. The partnership means that Liebherr and Microsoft are collaborating in the development of Liebherr’s new generation of the SmartDeviceBox. This is a communication module which fits into Liebherr refrigerators and freezers, connecting them to the internet. The system is designed to have a long lifecycle: Modular units can be integrated and upgraded at any time in existing SmartDevice-ready appliances to create value and comfort for customers through new digital features and solutions.

With the SmartDevice capability, future Liebherr refrigerators will help in shopping and planning meals with intelligent food management. Stored groceries can be monitored using internal cameras and object recognition technology. This process not only captures images for viewing but also recognizes individual food items inside the refrigerator. This information flows automatically into an inventory list, which lets the customer see quickly and clearly what is in the refrigerator from anywhere. Using the SmartDeviceBox voice module, additional groceries can be added to a shopping list that customers can access when on the move using an app for iOS, Android or Windows devices.

Microsoft is providing computer vision capability as part of this collaboration. As a leading innovator of deep learning algorithms for image processing, illustrated by our winning entries in the 2015 ImageNet object detection challenge, Microsoft’s deep learning technology is now the cornerstone of a wide variety of products, including the Microsoft Cognitive Services Computer Vision API which can – among other things – annotate new images with a wide variety of generic category labels and object tags. The underlying state-of-the-art deep learning algorithms themselves are also available within Microsoft’s open source Computational Network Toolkit (CNTK) and can be used to build custom models for new use cases.

Using the deep learning algorithms contained within CNTK, Microsoft data scientists worked with Liebherr to build a new image processing system to detect specific food products present inside a Liebherr refrigerator. Starting with a general purpose computer vision model learned from millions of generic images, the Microsoft system can learn to recognize new types of objects – milk cartons, ketchup bottles, pickle jars and much more – from example images. When a new image from inside a refrigerator is provided to the newly learned model, it can detect the presence of the objects it has seen before during training. The image result below shows an example image with labels of the food products detected (and confidence levels) overlaid.

Although the current system is a prototype, the deep learning technology it uses is already very powerful and rapidly maturing. Liebherr and Microsoft are actively improving these technologies to turn the newest generation of refrigerators into smart appliances that don’t just cool your food but interactively help you with your food management.