What started back in 2008 as a way to get developers excited about the possibilities of the Android platform, recently reemerged with a new global ask of the developer community. This time, the goal was to create helpful innovations powered by on-device machine learning (ML). This latest challenge, released in December 2019, set out to unlock new experiences on Android and help to inspire developers around the world, similar to prior Android Challenges over a decade ago.
Not ones to shy away from any technical challenge, a team of Bottle Rocket engineers, along with our friends at Google, collaborated to create a one-of-a-kind app that not only earned them a finalist position in the Android Challenge, but also solved a key issue for the visually impaired.
After just a few short weeks of concepting and a few months of development, the Path Finder app was born with a goal of helping users navigate through public environments and all the unknowns surrounding them. Leveraging the ML capabilities of Android, Path Finder helps people with visual impairments navigate through complex situations like shopping malls or crowded streets by identifying and calculating the trajectories of objects moving in their path. Custom alerts then inform the user of how to avoid these obstacles and what actions they can safely make.
“What started as a simple idea among Rocketeers was selected as one in a handful of finalists out of over 500 applicants,” says Colin Shelton, Lead Android Engineer II at Bottle Rocket. “Our ML club is always looking for new opportunities to flex our technical chops and experiment with technology in new ways and this challenge was the perfect opportunity to do just that while also doing something good for the world. We’re excited to watch the app improve over time as Android devices continue to mature and advance.”
The app brings together object detection and depth mapping and uses TensorFlow Lite to calculate distances of surrounding objects. It is designed to augment the user’s experience, share information, and give support, not overwhelm them in difficult situations. For this reason, Path Finder’s setup process is made to be conversational and is customized for users with limited vision, as well as the people helping them. Both audible and haptic feedback are part of the obstacle alert system, while a range of pitches and frequencies communicate each object’s distance and direction. Audio patterns, like Morse Code, are then layered and combined for sharing further information. The app helps visually impaired users gain the advantage of foresight, making public environments easier to navigate.
“The Path Finder app is an open source project that offers developers a starting point to take this concept even further,” states Luke Wallace, Director of Android Engineering at Bottle Rocket. “The Bottle Rocket team has effectively created a proof of concept and validated what is feasible through on-device ML. Our teams have proved that this technology has real world purpose and can truly help those in need. Path Finder is our gift to the Android developer world and the community.”
The app has been released to the public and is currently available in the Google Play store. The Path Finder app is also featured among other finalists in a curated ML collection by Google. Google plans to leverage technology influencers to spread the word about each of the finalist apps to help promote both the technology behind the app, as well as the apps themselves.
“I am so impressed with this team of engineers and other Rocketeers who worked hard to produce the Path Finder app,” says Calvin Carter, Bottle Rocket’s Founder and CEO. “It’s proof that a small group of committed and passionate people all leaning in the same direction can achieve big things together. It’s also a perfect example of ‘love of the game,’ something we talk a lot about at Bottle Rocket. They did this work because they authentically care for others and out of a true desire to learn. I recognize you and thank you for this amazing contribution not just to the men, women, and children who will be able to use this app to better experience the world, but also making the world a more inclusive place.”
The Path Finder app can be downloaded here: https://play.google.com/store/apps/details?id=com.br.ml.brpathfinder