Analysis of medical images is essential in modern medicine. With the increasing amount of patient data, new challenges and opportunities arise for different phases of the clinical routine, such as diagnosis, surgery and therapy. The InnerEye research project focuses on the automatic analysis of patients' scans. It uses state of the art machine learning techniques for: Automatic detection and segmentation of healthy anatomy, as well as anomalies; Semantic navigation and visualization for improved clinical workflow; Robust registration for monitoring disease progression; Natural user interfaces for surgery. Our methods combine medical expertise and machine learning theory in the design of tools for computer-aided diagnosis, personalized medicine, and efficient data management. These tools are being incorporated within Microsoft Amalga Unified Intelligence System.