Analysis of medical images is essential in modern medicine. With the ever increasing amount of patient data, new challenges and opportunities arise for different phases of the clinical routine, such as diagnosis, treatment and monitoring. The InnerEye research project focuses on the automatic analysis of patients’ medical scans.