Abstract Breast cancer is the most commonly diagnosed cancer among women worldwide. Patients diagnosed with breast cancer face the important question of what therapeutic regimens to choose, and they are eager to know the effects of potentially applied treatments. While treatment decisions have become more refined over time, they are not personalized and as such patients may still be over or undertreated. There is a paramount need to integrate the multi-scale, multi-modal, and multi-timepoint patient data and to build the capacity of systematically and accurately assessing a patient’s individual data to guide precision treatments of breast cancer. The complexity of multi-modal datasets poses challenges for physicians to interpret and integrate information, where artificial intelligence (AI) and data science are capable of extracting, aggregating, and inferring predictive insights. The goal of this study is to develop ethically designed AI prediction models using multi-modal data (clinical variables, medical images, and genomics assays, from individual, macro-scale, to micro-scale and longitudinal) to assess treatment efficacy of breast cancer and guide precision treatment decision-making. We propose establishing a multi-center collaboration network (University of Pittsburgh, Duke University, and MD Anderson Cancer Center) to curate diverse patient data for the AI model development and evaluation. We have assembled an experienced multi-disciplinary team with data scientists, oncologists, radiologists, geneticists, surgeons, pathologists, biologists, and biostatisticians. We propose to study two specific aims to demonstrate the concept by year 2 through establishing the team and collaboration network, delivering AI models, contributing new AI techniques, and crafting plans for continued work.