People are increasingly conscious of their everyday health and wellness conditions, and actively seek to improve them. In this project, we build software and hardware solutions that utilize and/or augment mobile phones to continuously monitor users' wellness without changing their existing lifestyle.
Our system consists of HW/SW, apps, Cloud and social networks and features a closed-loop design with both sensing and actuating capabilities. In particular, instead of solely passive monitoring, we further explore the actuation possibilities, i.e., seek to leverage the social networks to properly motivate the user towards improved health conditions.
The Hardware Platform: Septimu
We redesign the earphone as a sensing and controlling device by integrating sensors into the phone buds. The earphone is perhaps the only widely accepted accessory, and is standard on all mobile phones. Therefore, turning them into controllers and sensors will have a huge impact on both the mobile gaming industry and mobile healthcare. Initially, we integrate a thermometer, and two IMU units into the earphone. We reuse the audio jack for communication purpose. With this sensory earphone, we can demonstrate a number of applications, including posture detection, health diary, exercise patterns and coaching, and remote doctoring.
Health Lifestyle and Diary: continuous in-situ monitoring allows us to monitor user head posture, analyze exercise patterns and automatically generate a “health diary” for the user. We use SEPTIMU to monitor various body conditions while the user is listening to music. This approaches allows to collect data naturally and without affecting the usual lifestyle of the user.
Mood Sensing and Affection: this application collects a variety of sensing data, including speech, activities, heart rate, location etc to detect the user's mood and mental health. It also explores the power of music and study the affectiveness of different types of music for particular moods. It further leverages the social network to recommend music to users, given a specific goal.
MusicalHeart: A Hearty Way of Listening to Music:
We leverage Septimu and develop a convenient, non-invasive, personalized, and low-cost wellness monitoring system. With the IMU sensors on the earbuds, we obtain fine activity level of the user; using the sound signal from the internal-facing microphone on the earbud, we reliably detect the heart rate even the user is listening to music. Such physiological and activity level information are then used by an intelligent application on the phone to recommend, play and share health and situation-aware music. In particular, we demonstrate that we are able to to assist the user to maintain a target heart rate under different activity levels through dynamic music suggestions.
* Joint work with my intern Dezhi Hong (now at UVA), Xiaofan Jiang (now at Intel) and his intern Ben Zhang (now at UC Berkeley), Feng Zhao and Prof Jack Stankovic (UVA, US) and his team.