Efficient Bayesian Detection of Disease Onset in Truncated Medical Data

This paper describes a principled statistical method of preprocessing incidentally collected electronic medical records to facilitate short-term predictions of disease onset without explicit interaction with patients (e.g., medical tests, question- naires). The model is also applicable to detection of remission. In incidentally collected data, records are possibly left and right truncated the first time an event of interest is seen in a patients data may not be the first time in the patients history that it happened. It is therefore difficult to know if a disease onset happens in a given history. If we are unable to determine if and when the onset occurs, supervised learning and regression approaches cannot be applied. Our method determines if an onset occurs in a set of sparse and incomplete patient records, calculates the time of this onset and provides a principled measure of confidence. It combines individual patient history with expectations computed from a reference population. We compare the proposed method against standard change detection algorithms on generated data with realistic event sparsity and show that it can reliably detect onsets where traditional methods fail. We then go on to apply the algorithm to a large corpus of U.S. Medicare data and show that the algorithm scales to large data sets efficiently. The algorithm is currently in trials at a large medical informatics company.