PCEC News
Mobile App Helps Collect Data for Training Whirlpool Smart Appliance Machine Learning Algorithms
January 17, 2020
Whirlpool’s smart appliances are revolutionizing the way everyday consumers interact with their household products. The objectives of Whirlpool’s Smart Counter Top Oven include a breakthrough user experience, enabling relationships with consumers through assisted cooking, creating a focus on results-based cooking, and step removal through food recognition. Sponsored by Whirlpool, GVSU computer science students Justin Perticone, Marshal Brummel and Aron Ockerse collaborated closely with ADI/Whirlpool Corporation’s Dayanand Suldhal, Lead Engineer and Tristan J VanFossen, Analyst Engineer. The team was tasked with creating a native mobile application that will allow internal users at Whirlpool to contribute to the data collection process of the project by annotating images of photos of food. This information will be used to train and improve automated image recognition algorithms that will automatically recognize food when placed in the counter top oven. The app incentivizes participants by awarding them points for their efforts.
The team used React Native to implement the app. React Native is a JavaScript framework that is rendered with native code. This means that components in React Native, such as Buttons, are rendered using the native iOS and Android platform code to give them the native look and feel on both platforms. There are significant advantages to using a framework like React Native for developing native mobile apps in that it provides a single codebase for both platforms. That is, the app is implemented once in JavaScript and then automatically translated into binary installable app bundles for both the iOS and Android platforms. Implementing the apps directly in the SDK’s provided by Apple and Google require developers to develop two different versions of the app (typically one in Swift and the other in Kotlin!).
Users can use the app to take pictures with their camera and submit it. Each image that is to be submitted, must be associated with one of Whirlpool’s ongoing campaigns, before submitted. Once submitted the images are stored in Firebase, Google’s scalable cloud backend.
One of the team’s primary goals was to create a simplistic, user-friendly, intuitive UI across all screens in the apps. The main goal of the app is to help facilitate the capture of image annotations to help improve Whirlpool’s image recognition algorithms. A user is presented with a list of images that are eligible for annotation. The list displayed is filtered to prevent a user from annotating his/her own image, and also to make sure a user never annotates the same image twice. Once a user selects which image they would like to annotate, they will be brought to the second screen of the annotation process. It is here that they will annotate the image and submit data to Firebase. Once the image has received the specified number of total annotations, this screen will begin to display the top three most commonly received annotations for the selected image in alphabetical order. This is done to avoid early bias and encourage consistency later on while streamlining the annotation process.
These sorts of experiential learning opportunities help GVSU computer science students practice concepts they learn in the classroom, and also develop professional skills experientially, by collaborating with an industry partner.
“It was a good experience to develop our application using a SCRUM methodology”, says computer science student Justin Perticone. “Breaking down work cycles into two week sprints helped maintain project organization and keep on task. In addition, reporting and presenting our progress at the end of the sprints to our peers helped us further develop our communication skills.”
“GVSU Senior Project initiative is an excellent avenue to bridge academic world with industry-based experiential learning. The GVSU team understood the problem and independently executed it with agility within the time frame” said Suldhal.
“They showcased the skills they learned as computer science students, brainstormed with our team and were diligent about demonstrating progress at the end of every development sprint. It was a real joy to work with them on this assignment!” added VanFossen.