Turing Lecture: Transforming medicine through AI-enabled healthcare pathways

today's lecture is in the field of health and medical science we're in an appropriate building and I'm very grateful to the Royal College of finishing physicians for what post in it here gallant Royal Institute is the National Institute for data science and AI artificial intelligence and these are the fields that often said that I suspect now rightly so to represent the core of the fourth Industrial Revolution transforming the way we will all live and work and this is certainly true of Health in terms of its transformational possibilities this kind of transformation is brought about by combining new and better data so-called big data with domain knowledge the clinicians and medical scientists and the rapidly developing expertise in AI and notably that machine learning so in healthcare this means presenting all the relevant data for a presenting patient the knowledge of the clinicians and the methods of mathematics and statistics combined into machine learning so we can actually identify three phases firstly assembling the patient's data data wrangling wholly non-trivial that secondly the machine learning algorithms which provides potentially rapid and precise diagnosis and prognosis offering augmented intelligence AI and another if you if you take the initials of that offering augmented intelligence for the clinician and thirdly the third phase the treatment plans if we can monitor the treatment plans and I suspect of the phases they're all difficult but I think this is possibly one that is most challenging we can combine the intelligence from that evaluation as new data into feedback loops then generate a learning machine so we can do this in embryo and in particular cases progress is being made for example in the Institute's work with the Cystic Fibrosis trust it's a huge challenge to put this into a practice at scale and to complete this loop and hailer and gem may have views of how long it's going to take my Helen once said to me this is 20 years Roy but I suspect it might be 10 years Roy or less but we will see the children in situ research is seeking to help to meet this challenge I should say it is substantially supported by the UK RI and strategic investment fund and this evening will be here in the first fruits of this so what I've said is that we need to combine data clinical expertise and machine learning and these are fully represented with our two speakers tonight dr. john ross bus is the national director for disease registration and cancer analysis in public health England where it has 350 staff in the national disease registration service and it's probably a useful point where I can interrupt my introduction of the speakers by saying that the alan turing institute is distributed it works with partners it has to work with partners we have 13 university partners and we have a range of partners in industry and the public sector so a partnership with gems register fund put it that way and public health England is extremely important to us professor Mahler Hunt Thunder char hits it on the screen is the John Humphrey Plummer professor of machine learning artificial intelligence and medicine in the University of Cambridge and she's a Turing fellow at there on tauron Institute so the title of tonight's lecture is transforming medicine through AI enabled healthcare pathways and I'm delighted now on your behalf to invite German mihaela to give their lecture jenna is going to kick off [Applause] Island thank you very much good evening ladies and gentlemen it's a great pleasure to be here what we're going to do over this hour is to take you on a journey and it's an extraordinary journey about healthcare that we couldn't be in a better place really that here we are in the Royal College of Physicians that in 1618 produced this unifying pharmacopoeia for how clinicians should practice but medicine is hard medicine is difficult and medicine is hard because it how do you treat the patient and how do you identify and diagnose the patient and our traditional way of doing this has been to use medical histories what the patient told us the examination of the patient investigations and then to gather that information together and on the basis of that what we knew from text books what we know from guidelines what we see in published papers and what we know in trials to decide on how to treat that patient and that method is largely unchanged the problem we've got is the amount of data we're getting is greater and greater yes and this for anyone who reads the medical literature will be familiar this type of diagram which is you know a clear representation and this one is the phospho notes at all pathways it's telling us about pathogenesis about disease and how it happens and it's extremely important in understanding disease mechanism it's important in telling us about how you classify disease and how you might do therapy you take interventions but we still need to treat the patient sitting in front of us and it's only once we've made the diagnosis and we aren't begin to understand their disease that we can do treatment so the challenge is really how to deliver personalized medicine for each individual and that is the challenge that we face today as patients come in to see us in clinic we're good at pathogenesis we're good at knowing more about inherited risk but inherited risk doesn't tell us anything everything in in the case of cancer five to ten percent of cancers have a clear hereditary basis everything else we find when we examine the tumor later on the germline mutations are not there we know about exposure issed we know the demography comorbidities but bringing this information together is the key to understanding how we treat individual patients and that is the challenge for us this stack of needles is that we've gone from seeing patient of trying to understand the patient that is the needle in the haystack to us all being different that we're essentially now a haystack of needles and that idea that every patient is different who our exposure our lifespan are the interventions to pathogenesis there are comorbidities all of that information needs to be brought together to allow us to know how to diagnose and treat a patient and that is the challenge the second challenge is then bringing that data together but then having the methodologies and processes that allow us to interpret that data and make sense of it and that is what machine learning is being able to do is do for us the first place to start is yeah we've got genomic Gage and when I trained in medicine 20-30 years ago you know the genomic resolution was still to happen suddenly we thought we were going to understand medicine completely because we were gonna have three times ten to the nine bases on every patient and every person and everything would become clear it's a good Big Data problem I see T&G basis of make up DNA and the complexity of 3 times 10 to the 9 is a complexity problem that is a standard Big Data one the clinical one is completely different the clinical one is of a hundred and fifty different clinical vocabularies of five million different terms of a data complexity problem where that information even when a doctor says you've got pneumonia what did you mean when you said you had pneumonia yeah so even the terminology we put in becomes a problem so the challenge that we face on this side for clinical medicine is a data complexity run on this one is a big data problem what I've been doing over the last 10 to 15 years is trying to solve this particular problem on clinical data how do we produce a high resolution longitudinal data set on every person that we can use it as a resource to drive what we will show you later today is really personalized medicine and we pick cancer to start with cancers an interesting condition but well is in one condition it's this vast complexity of different diseases but the beauty of cancer is we probably understand personalization in medicine in cancer more than in any other particular discipline it also spans everything from hereditary risk through screening programs different interventions through treatment types through long-term outcomes we know a lot about the morphology so cancer was an interesting place to start not least because actually when you want to go after data the cancer registries were the place to start probably best described as the Jurassic Park of healthcare data cancer registries were the sort of land that time had forgotten they were a group of organizations that were set up with the founding of the NHS that were responsible for protecting the population from radon exposure and simply they contain people who went after every case of cancer in the population and they would collect data into this organization for incidence mortality and survival but were the important thing was they wanted to find every case of cancer in the entire population so I decided that the place to go to find this data was the Cancer Registry the other advantage the cancer registries had and I had been involved in various government initiatives to try and pull data from across the service is that this was a difficult problem I mean as Alan has just said you know this is a 10 to 20 year a problem a lot of what we try and do is much more short-term but getting data out of national health systems out of any complex health economy is extremely difficult it's going to take years and so the cancer registries were a nice place to go and as someone who works for government I learned what the word well-established means well established is a civil service term for boring and actually being boring is where you need to be you go somewhere that is quiet that you can spend trying to stall what is a difficult problem which is how do I get record-level identifiable data on every patient of cancer across the whole care pathway and then curate that to sufficient quality that I can use it for what we now want to do with machine learning and that was the challenge and that has taken us 15 years but it's not just a data set the importance of this is building a whole care pathway when we see a patient in clinic it isn't just the person we see with a particular symptom then it's understanding the whole of the history the exposures the other treatments the whole of that lifestyle the whole of the other comorbidities that patient may have billed what we're interested in which is pathways and a care pathway is a a collection of that entire healthcare event over the entire lifetime of the patient which you can then continue to follow in near-real-time I'm not building an electronic patient record system I'm not building things in acute hospitals what I'm building is a secondary data system but high resolution high quality for the entire population and at very large scale the challenge we have with the haystack of needles is how to find similar needles and as we take even the most common of cancers like breast cancer 44 46,000 cases a year in this country by the time we've sub fragmented those by histopathological type by more four by molecular subtype by comorbidities by other patients we're into a very small cohorts of individuals so even the most common conditions begin to break down into relatively uncommon relatively rare conditions so you need a data set to be able to do that but you also want to move away from a traditional registry it isn't just a stamp collecting organization that says we collected a lot of data you really want a data service a system that can monitor healthcare in near-real-time across the population which watch healthcare across a range of organizations that understands the flow of data and the beauty of that is you then have something that you can begin to put interventions into you can put algorithms into a into a service system and you need the third thing which is the implementation arm to be able to take anything you do back into the service and that is what we've set about to build and so we've moved away from a standard cancer registration system into a registration service won that contract patients over their entire care pathway in near-real-time its services that will drive what we want to do as an idea of scale this is what the UK looks like 56 million population primary care centers radiotherapy centers and my aim is to get data across the whole of these and so we played a number of games to try and get this data it's difficult anyone who knows anything about healthcare data systems anywhere in the world getting data out of healthcare systems is difficult because of those 150 vocabularies those five million different data items so what we did was we actually broke some really quite fundamental rules one of the first rules we broke was we abandoned data standards one way to approach this is to say everyone must do the same so we're going to force you to change your data systems to our universal standards so we can collect your data my experience of doing that is it's hard and it's hard because you it takes you probably two to three years to agree on your data set and agreeing on that data set even after three years is only eighty percent of what you wanted because you couldn't agree on the last and at that point your problems only begin because actually you've still got to implement it in all of the systems and what you've produced is probably obsolete but if you turn it around the other way and it's out you want is information on patience just tell me about the patient their name address postcode date of birth I have a legal gateway to know this I can know about what treatment they've got and then you bring the data in and you make it so easy for people to send you data that it's embarrassing not to you can begin to get it out of organizations and that's what we built so we built systems to pull data across this whole care pathway for all of these events including the screening programs for genetic risk I don't have a legal gateway because the person does not have cancer yet but I can take the data in a suit on a nice form I can de identify it in a way that I can never know who that patient is until they get cancer when they get cancer I can create the pseudonym on my cancer patient and then match the previous history and we're already doing that for all the women in the UK who have genetic risk of Braca we can take all of the variants out of the testing clinic we can sit and watch those variants of unknown significance and we know which ones then go on to get cancer and we do that across the whole of this pathway yes pulling data on germline data screening data comorbidities MDT discussions 2,000 MDT's every week we also abandoned the format send us it in any format we will take the format you can send it to us most easily so you send us a full text pathology report I used to be a pathologist stage to generate this verbage it's highly valuable but difficult to extract it's not structured in any way at all we built a markup system in a special markup language called gamal which stands for yet another markup language and allowed us to take this data set and bring it into our data set for molecular testing for example the lab down the road university/college can do this send us PDFs we pull the data out of this so data is pouring into this system is being curated and we're obsessed by data quality because we're aiming to deliver for personalized medicine and if you're going to deliver for personalized medicine for those even the most common cases you've got to be obsessional about data quality you've got to understand the stage the the treatment data to a degree of accuracy is is about 0.2% field level accuracy you cannot achieve that with natural language processing at the moment so there is no automated system that gets you everywhere and I have a lot of cancer registration officers who quality assured data but this is what I now get every multidisciplinary team 2,000 of them a week send me details about cancer patients and we build up this data set in 26 different sources around three or four hundred data items per patient we're now producing around quarter of a billion clinical items data items a year up from half a million in 2011 we then build that into a pathway and I want to take you through in the next few minutes is simply what a pathway of patient care looks like we've got at a system level this is a person level each row here and there are a hundred patients here is an ovarian cancer patient where the orange line is their major surgery at the date of diagnosis showing you one year before and one year after and there are a number of health care events on this pathway we don't have the complete electronic health record but we have a huge amount of data at individual level that we can join together and analyze and that's the raw is also that mihaylo and I have been working on its pathway data in a system that allows us to have that result so to just to show you what this looks like for lung cancer and the system we can now look at we can select any tumor site we can select the variety of those sites we can look where the trust where people were first treated this is mechanical data of co patient flow across the system but we can get down to a granularity that gives us all of these data points and so we can pull more and more data into the system and each one of these then gives us each this is a stacked bar chart of every healthcare event of nine thousand five hundred and twenty three cancer patients with lung cancer simply sorted by the length of their pathway I can drill into each one of those paths look at every event on those pathways join together quality or short and I have all of this data as my rule resource that is running but this system is a service it's coming out of healthcare systems across the whole of England every day so we can begin to populate this it takes us time we you know it takes us six months to get the data in and linked but we can get faster with newer technologies and those sort of things we're talking about is how we put in automated curation engines that allow us to do some of this but this is watching healthcare across a whole country and we can drill down into different morphological types we can look at outcomes from those and we can track the pathways of individual patients through a simple Sankey diagram showing the flow of patients through a system where we can now begin to ask questions about individual flows of patients of patients going into the system we can track cohorts of patients we can track crucial decision points such as this the multidisciplinary team meeting where data coming in of patients coming into the MDT we then see a range of discussions and processes and we've watched those going out we can pick data up from all of those places so in our system implementation not only if we got data out points we've got the places where we can feel send feedback into the system but where we want to go is straight back into clinic I can show you organizational level but ultimately I've got the patient sitting in front of me and this is the challenge of delivering personalized medicine for personalized medicine the patient comes into the clinic and suddenly I need to understand that in heart history and their pathway and this is a single patient this is our data set brought out for this individual patient showing the patient pathway here in the centre going back to 1998 with a series of events we can make this richer and richer this is a 73 year old woman with carcinoma the bronchus Stage four I also see a whole range of other specialties so I see all the other specialties that this patient's been treated in because as we get to our models and we begin to look at the analysis as this pathway what we want to be able to do is look at competing risks although we're interested in cancer the patient has a whole range of other diseases that we need to understand and balance and so this model that we build for cancer can be applied to every other disease we can know about the radiotherapy we know about the chemotherapy know about outpatient treatments we can work with industry to look at high cost drugs we can begin to do a whole trial analysis on every patient treated in the population with an individual resident and watch the follow-up there and we can drill into the data behind it so what we have in it here is a national registration service we're collecting data and patient level in near year at real time we have uniform high quality data curation and linkage and we can apply algorithm centrally to individual patient level data which we can then feed back into the clinic the crucial thing that we want to be able to do is the ability to deliver decision support direct to individual clinicians and patients and that's what Michaela and I have been working on so to get an idea about what we can achieve we would like to start by showing you a demonstrator which we have built in the past couple of months by my team the ML aim team this is the name of my research lab machine learning and artificial intelligence for medicine together with John and what you see in here is an ecosystem that can be used as a decision support system for a variety of diseases but but we would like to show to you first is a demonstration on breast cancer so this demonstration the important thing to recognize with this demonstrator is that this isn't although you're seeing a couple of screenshots taken of webpages this is a real demonstrator that's used our data yes this isn't just graphs we've drawn these are the algorithms that you'll see are running behind it the data set is the national disease registration data set that we have on cancer in this country and you're not going to see any patient identifiable data don't worry I'm protected for that but this has used this real data set to do this so the place we start is putting some diagnostic information in you know we've got input information about the diagnosis about the patient themselves and we've got information on the pathology we can pull the information from diagnoses from any patient you can go into the registration system and say pull me this patient's data and I get some basic demographic data coming out of the system what I can also do though is get the pathology data in now you saw the pathology report we used to write a nice full of words it's as opaque as it possibly can be that's called a profession that's why I'm a member of the Royal College of pathologists is I know how to write a report to make sure that people can't read it but actually machine learning and technology and string recognition is better than we'd hoped and so we can now take our full-text pathology reports any of them we want upload the Reposado G report and extract the data from what I try to make opaque into something that is really useful from the report and that gives us a huge power to be able to move forward so with those two data items we can then do some work yes so a first step that we can do a diagnosis time on the basis of the data that Jen has shown is to issue predictions of risk for this particular patient and we are doing that on the base of the technology that I'm going to touch upon called Auto prognosis what you also can see is that on the basis of this particular patient's we are when we are issuing such a prediction we not only issuing this prediction but also we are able to tell which variables are important then which variables are important for this particular individual patient in issuing this prediction so these are not feature predictions at the population level but rather an individualized level and we are able to do that with the technology that I am going again to briefly tell you about that was developed in my lab called in vase note also that these variables at the individualized level are varying and their importance over time so we've got individual patient data here but of course what we've got here is a patient who is issue receptor positive of course we can then change the variables that the patient has and here we have an ER positive stage patient let's say we turn them to Ernie our negative patient Auto prognosis we'll take that data and give us a new predicted risk and mortality risk over time so obviously the data is in there and it's you know providing us with new in but the other thing and again that I brought to you is that we've got an event based model here we've got the whole care pathway of these patients this is the prediction at the time of diagnosis but after diagnosis the patient leaves clinic other things happen other events occur and here we've got an event listing on this side of the screen showing the events that have happened to this patient so what we've looked at here is our prediction at time zero at the time of diagnosis two-and-a-half years later that patient comes back to see you two and a half years would be here but we know lots more about that patient we've done other investigations and all of these new events have occurred so what we've done together is use the technologies that Michaela has in her lab to look at how those events change that model and this is what happens so we see that actually now at 18 months the world is completely different for this patient we weren't making bad predictions but we know a lot more and we can now predict differently and we're having a different conversation with that patient but the crucial component behind this is to understand how do we make this transparent is this something that is just going on behind the scenes or other ways of understanding where those events come from and for that we have developed a longitudinal model that is able to take these different events the duration of the different events the order in which these events are happening for the particular patient and issue predictions and adapt these predictions continuously over time and this is based on a technology that I'm also going to talk about called pass and so if I look clinically at those events yes a lot of them make sense but what I really want to do is perhaps understand Anna normally here yes you'll see there's a funny little peak here something seems to be going on there's a data event in this cluster of data that we've got that has suggested that you know there are estimated probability suddenly did this jump what is that date and where is it and the beauty of this system is we can begin to make that transparent so I can look in more detail at the range of properties that have contributed to this individual data point and say can we find which one it is and actually if I look here I've got another tumour diagnosis has taken place at a crucial point which I can now decide if I remove that what effect does it have on this curve and I can take it out and if I take it out I can see an effect so I can begin to understand at a person level at a data item level the contributions that these individual metrics that we've got contribute towards you know the the prognostic tools that we produced so what we are able to do is to understand what's the value of information and was the value of a particular event in the context of other events and other comorbidities that may happen and understand what's the value of information over time this could be used both to understand why a particular prediction has been made but also what other data to collect and to collect in the future as well as what monitoring should be done at the personalized level and we do that using a technology called deep sensing but we can go one step further we not only can make predictions about the patient in the short run and in the long run we can also make recommendations about what particular treatment should be given to a patient so what you see in here first is recommendation for the patient at the population level and what you can see is that if we are going to use population level information to issue these predictions you would think that doing chemotherapy and radiotherapy would be the best outcome for this particular patient and these are the propensity scores associated with it so one could do things like propensity score matching however using technology that we have developed in our lab you can go and step further and learn on the basis of observational data how to do individualized treatment effects recommendations what would be the treatment that will be most successful for this particular individual patient and what you can see is that in this case it is the use of chemotherapy rather than chemotherapy radiotherapy which would lead to the best prediction and I'm going to tell you more about that in a little bit and finally I'm sitting in clinic with the patient in front of me and what I want to know is how similar are you to other patients because that allows me to ask a whole range of questions even if it is one that what will your prognosis be what will the outcome be what how should I treat you how can I tell you what events would be but similarity is very hard to determine you can do it simplistically simply on the age and sex of the patient than tumor type it is good enough the beauty of the technology we brought to this is it allows us to do deep predictive clustering of individual patients where we can say based on all the features we've got in the model find me the patients who are most similar and that is the start point for everything else in medicine that goes back to why we're in the Royal College of Physicians that is the understanding of how do I treat the patient based on my knowledge of similar patients so now let me walk you through several of these technologies you have seen them in pink appear one after another I'm going to briefly tell you about some of them so how did we build such a decision support system that I described before a first goal was to learn trajectories of disease for a patient that are individualized and personalized and that enable us to understand how on the basis of information that we have a diagnosis time as well as overtime we can revise prognosis but also we can identify what treatments we should recommend to a particular patient and as this information is collected over time revised trajectories of disease for the individual patient and feedback analytics and recommendations to the patient as well as their clinicians to be able to improve their care a first question if you want to do something like that is to think about how to model in a systematic and rigorous way health and disease trajectories and how to track this is overtime in an individualized way so this gem has shown we have often different types of observation measurements symptoms or events that are available for a patient and on the basis of this what we would like to do is to identify what is the underlying state of health or disease of a patient and what ideally we would like to do is to do a much more systemic disease categorization that we have today so maybe cancer a particular type of cancer doesn't pass for four stages maybe passes for multiple stages and maybe different number of stages for different types of patients that may have different underlying genetic material but also which may have different type of comorbidities so how many states does a patient pass through from a health to a particular diagnosis how long does a patient take to change between the state how long will patient remain in a particular state and what is the probability to move to a new state depending on a treatment or on an underlying comorbidity all of this we would like to do it systematically and learn on the basis of data so what we would like to do as a first step is to infer not only the current state but also forecast what is the probability that the patient will move within a certain time horizon to a next state of interest will the patient move to this particular stage of cancer will it move to another state how long will it stay in that particular state and was the evolution in the short run in the long run this will also enable us to determine how to act and when to act should I wait and watch all the intervene now and what will be the influence on this from the trajectory of a patient so what we would like to do is we would like to define a regulatory state infer transitions between states duration of states at the individualized level and what we would also like to do is we would like to determine what's the probability to move to a next state potentially an absorbing state a mortality state either predicting risk due to a Maine cancer or due to a related disease or for instance in this case cardiac mortality due to possibly an intervention that was a result of the primary cancer treatment so what we are interested is not only to look at prediction of mortality or a prediction of recurrence or a prediction of a type of event but a variety of events so we would like to do that in a holistic way so a question that can come me as well we have had machine learning and artificial intelligence for quite a long time what why this technology is not used today and my answer would be that many of the existing technologies are inadequate and simplistic they are unable to capture the complexity of medicine there are one-size-fits-all uninterpretable and not easy for clinicians and medical researchers to act up one let us take a brief look at what is happening today in machine learning a lot of the existing methods are based on Markov models either Markov chains like for instance this is a very recent 2018 model of breast cancer and breast cancer revolution this is a very simple Markov model another example comes from the area of prostate cancer and examples like that continue more sophisticated methods are using hidden Markov models or more recently deep Markov models and usually the technologies that we have that have been used to model trajectories of disease are using some type of model which is again Markovian which compacts the entire past history clinical history of a patient only within a current state thereby forgetting what has happened to this particular patient while this methods these Markovian methods are easy to compute and understand they are often wrong history matters the order of events matters the duration of the different events matters and all of that is ignored in these models also these models are one-size-fits-all the state transitions the number of states are the same for the entire population independent of other comorbidities or independent on this particular person having a particular mutation so we'll deep-learning solve it all we hear a lot every day about how wonderful deep learning is so deep learning solve this problem well we hear a lot about deep learning methods such as recurrent neural networks which are capable to deal with longitudinal data such as for instance modeling language and making either that be having the ability to either do translations or to do outer audio recognition and indeed recurrent neural networks are very powerful but they are not powerful enough to deal with the complexity of medicine let me tell you where deep learning stands in medicine and I'm citing here a variety of works inclusively my own work which has been limited in the past to this understanding what we are usually doing when using recurrent neural networks we are applying a mechanism called attention to tell us when making predictions what events in the past are really important when issuing a prediction on the basis of the history and while the predictions that are issued may be somewhat correct and pragmatic they are not interpretable pathology we don't understand why something necessarily has happened how long this particular event is going to last and what will be the effect of a treatment if we are applying it so while recurrent neural networks are able to deal with the timing and order of events they do not capture this concept of state and they are not interpretable also in certain clinical knowledge from clinical notes or from icd-10 codes it's complicated to do in this type of setting and it doesn't allow us to answer important questions such as when this disease really started and what preceded it and how on the basis of this we expect that the patient will evolve in the future so what we really need to be able to do that we need models from machine learning that are clinically actionable and provide us with the necessary intelligence to issue prediction that are interpretable at the patient level we need to be able to learn from complex data like the type of data that gem has discussed able to learn from clinical annotation and medical knowledge we need to have models that are able to encapsulate the individualized histories of the patient effectively deal with multiple morbidities and understand that not all patients are the same as well as come up with interpretable models so a first clinically actionable model that has developed has been developed in my lab is called pass it really aims at solving a variety of the challenges that I mentioned before it is a general and versatile D probabilistic model that is able to learn on the basis of such non stationary clinical histories and it combines ideas from statistics with ideas from deep learning it is this wonderful synergy that gives rise to a variety of good solutions this model Pass is able to maintain the probabilistic structure of hidden Markov model such that we still can think about states but use recurrent neural networks and their power to model the state dynamics and be able in this way to do non stationary dynamic prediction so in pass the model dynamics are looking at not only the current state but the entire history of the past States and unlike existing recurrent neural networks which are just looking at events and events over time in our case we put attention and we look at how much a particular past state has contributed to the current state so unlike looking just events because the same event may happen in one moment in time and merripit itself it is this notion of a state that is really the one that matters and drives interpretability because one event happening in one particular state may have a very different impact on the evolution of the disease than the same event happening within another state so what you see in here with a little bit of an abuse of a notation of a graphical model is the patient is currently for instance in state six and what you can see on these weights is how much past states have contributed to the patient transiting in the state six so it is not only the past previous state like in a Markovian model but also states of the past that have led to the patient transiting finally to this particular state six so this algorithm pass is repeatedly updating their attention weights to focus on past state and is learning that on the basis of the data so this transition weights create if you like a soft version of a non-stationary variable order Markov model when the underlying dynamics of this model change over time and they depend on the individual clinical context so what drives these models is both static information acquired about the patient such as gender or genetic information as well as the evolution of the patient over time so the past memory are like in a recurrent neural network is shaped by the patient current context both clinical events as well as treatments and in this way we are able to move from a one-size-fits-all model where the state and state transitions are the same for all the patients to a contextual rotational mechanism where the attentive states depend on both the static information but also the dynamic evolution of the patient in this way this attention weights are providing us with the ability to explain positive and associative relationships between the hidden States and the past clinical observations for that particular patient what can we do with this so before that let me just in one slide say this is actually a very general and versatile model that encompasses a variety of other models that have been introduced over the years some of them in medicine some of them in other areas and highlighting here what has happened in medicine in my lab and elsewhere but the reason we need a very versatile model is to have the flexibility to learn from the data complicated disease trajectories granted it may be that for a particular disease we don't need a very complicated model and just a simple for instance continuous time T the Markov model we do you don't need a sophistication of path but this model is able on the basis of the available data to say a simple to state model would be sufficient for this particular disease to be able to understand it and act upon well for something that's more complex for a patient that has other comorbidities a more sophisticated model is needed what can we do with this then is we can issue predictions over time dynamic and personalized forecasts on how the trajectory of a patient is changing over time so we are moving from the current time to event analysis which is done often in statistical analysis to a dynamic and personalized forecasting where as events are collected about the patient new trajectories are predicted for the patient also we are able to do this time present analysis not only for something like morbidity which and death which is not really actionable but a variety of competing risks once the probability of this patient within a year or within five years to develop a cardiovascular event what is the probability for recurrence what is the probability that a patient that does have diabetes will develop cancer within this amount of time so we can look at competing risks as well so what we can do is we can move from the current state-of-the-art in epidemiology and medicine which is at the populational level we often see these beautiful network graphs that are done at the population level where it says these disease and this disease interact and that graph is at a population level we are moving now and we are able with these technologies to do a personalized morbidity Network we are depending on the patient characteristics this interaction graph is different but also importantly it is dynamic it is not once and for all as a patient is developing certain new morbidities or is taking medication is not developing a particular morbidity the graphs of morbidity a morbid interactions are dynamically changing over time and we are able to learn that on the basis of algorithm such as pass what else can we doing this we can determine personalized screening a monitoring profiles for a patient currently often when a patient is being diagnosed and treated for cancer it is sent home and it is asked to come back after a certain period of time and often this period of time is the same for all patients it is clear that depending on the projected trajectory of disease of this particular patient the patient should come earlier or later and the types of tests we should be doing on this patient may be different whether we should do an MRI at this particular moment in time or not so we'd like to identify what's the value and the value of information over time but also we would like to answer questions such as what modality should we always been doing a test for instance a radiological test what's the value of this particular test at this particular moment in time on this particular patient and we had developed technology to do that as Janice said we'd like not only to make predictions but also we'll be able to understand what our patients like the current agent and we would like to move from the static way in which we do phenotype into a temporal thermo typing paradigm where we are learning over time how patients are pertaining to a cluster and on the basis of that we can tell the patient this is what we expect for you to be the events that you are going to undergo in the next three months one year two years such that a patient can better plan their own health so this is both useful for clinicians as well as patients so what you see here is an example take from a real patient where initially a diagnosis time the patient was in cluster 1 and the probability of that actually was quite small and then upon a second event which is the response of this particular patient to radiotherapy the patient suddenly moved to cluster free and then after a longer time something else has happened that moved a patient yet to another cluster so it was on the basis of the events that have happened that the clustering is changing and this patient is uh pertaining to a different class as we mentioned in order to move from the current stage we are in where we are making just predictions you would also like to empower clinicians with tools with recommendations of treatment and treatment effects at the individualized level so what type of questions you would like to answer if bob was diagnosed with a particular disease for instance cancer what it meant would be best suited for Bob in the demonstrator you saw that for the particular patient that we show to you a different treatment was useful for this patient and for the general population so the problem you would like to solve is to develop methodology that is able to estimate the effect of a treatment on this particular individual and for that current state-of-the-art is mainly looking at randomized control trials but randomized control trials are at the population level they tell us about population level effects often they have small sample sizes and the patients that are enrolled are often not representative they may have no commodities they may be quite healthy younger patients but the patient that we need to treat may have comorbidities and may be older granted this is something still very important and there are many frontiers in this area inclusively designing better clinical trials and I am many others develop technology to be able to empower clinical trials but in addition to clinical trials not instead of clinical trials to complement clinical trials we would like to use the available observation data gem is collecting to learn what type of recommendation we may be able to give to this patient and how we can do that on the basis of observational data after a treatment has been approved and this is a very important area of research for for us how do we think about this problem we think about this problem within a very classical framework that was introduced almost a century ago by nima and this is the potential outcome framework where the idea is that we have a patient with unique features our X I we make a decision on what treatment whether to treat or not to treat this patient for the simplicity now I'm going to assume just a binary decision treat or not treat but I'm going to talk about multiple treatments in a little bit and then I observe what has happened upon that decision so what's important to note is that we only have factors but not counterfactuals so if we decided to treat we don't observe what would have happen if we didn't treat and vice-versa and what we would like to do is look at observational data and on the basis of the unique characteristics of the patient identify the effect of treating this particular patient to be able to do that we need to make to the assumptions one quite strong one weaker one strong assumption is that there are no hidden confounders and this is a strong assumption that I'm going to make granted we are doing work in that area we are trying to relax this assumption and there is some hope in this area especially when we deal with treatments over time another assumption that we made is that there are no patients that are either completely treated or completely untreated if that's the case that's not the interest in Scenario the doctors know exactly what to do they are treating everybody or not treating everybody but for the case there is divergence between them that's the case of common support and that's the case that's interesting to us so what we try to do we try to use observational data these dots in here patients that have unique features X and these are the outcomes of this particular patient so what you see here is two response surfaces one for the untreated one for the treated we do not have counter factors and what we would like to estimate is the treatment effect for a particular patients with features X note again this is not a simple machine learning problem this is not one of the classifications of cats and dogs type problems this is a much more challenging problem is a causal inference problem where we do not have labels we do not have counterfactuals so we do know what has happened to the patient given an action but not all that happened otherwise another problem that makes this causal inference problem of estimating treatment effects more complicated is that we need to not only model features and outcomes but also treatments and yet another one is that of the fact that we don't have at our disposal randomized control trials that are huge what we are learning this form is from the observational data that Jem is collecting for instance and that has selection bias the doctors are not closing their eyes and saying I'm treating you or I'm not reading you they are looking at the patient and on the basis of their own biases are determining what to do so we need to deal with the selection bias when developing methods there have been a quite large number of works in trying to address this problem in the area of statistics of biostatistics of machine learning and also econometrics but a big challenge with this work is that many of the existing methods are ad hoc and have no theory behind them so what we went upon was embarked in to a process on trying to develop develop a first theory for causal inference for identifying individualized treatment effects where we asked two key questions the first is what is possible in the limit not what algorithm but what can we achieve given a particular data set with its own bias and its own complexity of patience this is the first question it's an information theoretic questions about what's achievable and then empowered with this insights we are trying to develop algorithms that try to achieve or get close to these bounds so we are casting this in a Bayesian framework again by combining ideas from machine learning and statistics and what we do is we have these two surfaces to respond surfaces of the untreated and treated population and we put priors over these two surfaces and what we are trying them to determine on the basis of the observational data is point estimates on the treatment effects and the focus is to develop techniques and ways of thinking that enable us to reduce the error between the two treatment effect and estimated one we are doing that through a loss function or an error function that we call precision of estimating heterogeneous effects or for hot death so what can be achieved in the limit our synthetic aliy that is computed and estimated through a minimax estimation loss so this is an error this is the error we are trying to estimate and trying to minimize and we are trying to determine the best estimate of the most difficult response surface if we are able to do that if you are able to estimate that will have an upper bound we have an information theoretic upper bound that can tell us for a particular data set that has certain characteristics how far can we go is there hope to estimate effects of treatment what drives this is two quantities two fundamental quantities one is that of sparsity how sparse are these two surfaces how few variables really drive these two surfaces how many features really matter when I'm looking at a response of a patient and the other one is of smoothness how much it fluctuates how rough are these two surfaces the future population and we are modeling that by looking at holder continuity or Lipschitz continuity these two metrics of sparsity and complexity smoothness are giving the complexity of the two surfaces we can then start to write theorems like this one and then we parse this theorem for you what does it say it says that asymptotically how well we can do an estimate in treatment effects from observational data the individualized personalized level depends on the response surfaces and their complexity the number of relevant dimensions and the smoothness of the surfaces and in fact it is the most complex of the two surfaces that will define the hardness of this problem but maybe unsurprisingly you can also see that there is no bias here and this is unsurprisingly because if we have a very large number of patients if you think I seem totally bias doesn't matter so should they not care about bias well in the high sample regime we do not need to worry about bias it is the smoothness and dimensionality that matters but in the low sample regime we need to worry about bias it is this offset that starts to dominate and this offset is nothing else than the rainy divergence between the two distributions which is actually nothing else than the selection bias so what does this say it says that in the low sample regime you need to worry about selection bias what in the high sample regime you need to worry about the response surfaces and how complex they are a quick distillery now we can learn and develop algorithms algorithms which in the low sample regime are able to effectively deal with selection bias by sharing for instance training data between the two response surfaces and in the large sample regime built flexible models that are able to model the complexity and capture the complexity of the response surfaces just very briefly one particular technology that has been doing that is by on Gaussian processes is a technology that I developed a couple of years ago and the idea here is that we have Gaussian process priors that are indexing the to response surfaces thereby being able to develop shared representations that are effective across the two population treated and untreated and we are able to learn how to matically with features are the ones that are relevant that are mattering for the response for the treated or untreated population whose automatic relevance determination and more details about this can be found in some of our papers additionally when we issue predictions we not only initial predictions we are also able to tell something about the uncertainty associated with these predictions which is important when we issuing predictions or recommendations for treatment effects what we showed you in the demonstrator was yet another technology a technology that was able to look not only at treated or untreated population but many treatments a multitude of treatment like often is the case in breast cancer so this was issued with a technology based on generative adversarial networks called g'nite but this is about treatment a treatment effects at the time of surgery and making decisions one time ideally we would like to make decisions over time about both chemotherapy and radiotherapy and we would like to understand what treatment plan to use when to start it and when to stop it so what you would like to do is to issue and determination of treatment effects and counterfactual outcomes over time thereby being able to determine what treatment would be the most effective when should it start and when should it so we are doing that through a counterfactual recurrent neural network with a technology that we have developed that is able to do such treatment effect estimation over time and we are doing that by building treatment invariant representation using domain adversarial knowledge and training and then predicting counterfactual using a novel sequence to sequence architecture that is able to determine the predictions of the different types of events that would happen under the various type of alternatives but this is for one disease ideally we would like to do this not only for breast cancer we would like to do this for colon cancer and many other cancers and in fact we have started together with Jen to look at a variety of other cancers but we would also like to do that for a variety of other diseases that the same patient or other patients may have and as a matter of fact one of the birthplaces of this particular technology was in there of cystic fibrosis where we first developed out of prognosis so what you would like ideally to have is to have technology that is able to operate at a click of a button for a clinician or a medical researcher and that we don't need to really reinvent the wheel again and again how can we do that we can do that by using machine learning to do the machine learning so rather than replacing clinicians from ashen learning which is what I often hear what we are working upon is something quite different we are trying to use machine learning to replace the machine learner so we are trying to replace ourselves by building machine learning technologies that can do such risk going and such systems at scale how did we do that we do that for a technology that we call Auto prognosis that takes data and then builds automatically pipelines pipelines that with data imputation feature processing classification and calibration as well as learn how to build algorithms across these different stages after such pipelines are created with aggregate and foreign samples that issue both predictions as well as explanations as I mentioned before this is quite interesting so okay so we have done this for a variety of diseases both in the area of cystic fibrosis in the scientific reports as well as more recently on the basis of UK biobank for cardiovascular disease but we are able to do automated machine learning the 24 predictions also for time to event analysis or survival models also we are able to do that for competing risks as well as data over time we are also able to do that for treatment effects and this is actually a technology that was highlighted in the top wall review as being a key technology that can empower clinicians and medical researchers to do predictive analytics at scale but we don't want to issue any predictions and confidence interval associated with it we also want to issue explanations we want to understand why a particular prediction has been made and we have been working very hard over the last couple of years to produce such technology let me tell you just briefly about one of them so we have plate box models like the deep neural network or an auto prognosis model but this is not understandable to a human being so what you would like to have instead these white box models we would like to have 16 mathematical representations that can be expressed for simple equations that a human can understand and interrogate how do we do that we do that using symbolic metamodeling so what do we do is we take a blackbox model created by a deep learning model or auto prognosis and we build a model of that particular model G and this G is a 16 mathematical representation from a class that can be selected by the user it could be a polynomial it could be the class of algebraic functions cross form solutions or analytical solutions and then what we have is we are creating the metamodel the Dex approximates this black box model how do we do that we are doing that by using a familiar functions the Meier G functions which are univariate special functions that are able to reduce to all familiar functional forms what we do then is we are projecting this black box function within these familiar functions and we are parametrizing this space this understandable space using a parameter theta and then for every setting of this parameter we are able to recover a new symbolic expression and we are going through a process of approximating the black box function through gradient descent to learn a white box function how do we do that by computing computational graphs that enable to do this effectively let me tell you what this means I have a breast cancer data erasure prediction like the one jen has shown to you photo prognosis or for an algorithm such as XG boost what you can see is then that such a machine learning model does better than state-of-the-art technology that is interpretable the predict algorithm so you'll have a better prediction but is this understandable well what you can see is that using our symbolic metamodeling we can tell you the equation associated with it so this becomes interpretable and what's important is that in this way new interactions become uncovered we are going to understand in this way the mind of the machine and what it discovers in this case is that nonlinear interactions are very important and what you can see here is if we use not the black box but the interpretation the white box model the performance that you get is very close but if you insist to have linear models only not interpretable by linear models you lose a lot in performance so what you have in this technology is have the power of complex models that are in addition understandable and you are able to uncover interactions that have not been seen before you can also go one step further and look at how variables may matter differently for different types of patients over time such that you move from population level importance of features to individualize predictions of injuries but this presents what we have done so far where are we going to go next so the challenge is to get this back yes we've got its extraordinary algorithms we've got extraordinary data systems we've got a service that is linking data together what we want to be able to do is real-time personalized decision support direct to individual patients and clinicians so the aim there is to be able to deliver something of scale and we have drawn on the word Michaela has done previously with others to look at how we would build a system that instead of just looks at data at rest allows us to do a much more dynamic process of real-time data as it is changing that is driving the machine learning model deriving the decision support and driving the process in order to be able to do that we nailed to pull data out of national data streams pour it into a machine learning decision support system and feedback actionable intelligence to individual clinicians and healthcare systems this is the work Mikayla's group has done previously in partnership with IBM we're already beginning to build this model and we have a view of how it can be applied both the data extraction systems that I've been building across England and the algorithms that mahalia and the team have produced that we'll be able to sit above the NHS above clinics and feed that information into the system so that once we've done appropriate validation with patients and clinicians we've got a formal accreditation and safety we'll be able to deploy this into the NHS our longer-term view is that this needs proper integration with the electronic health record and that the model that we can potentially produce is one that we can apply internationally so that if you've got an epic system or a Cerner system that the model will be to take the patient way out of those systems in an anonymized form pass them into the algorithms and systems that we have developed here and provide up-to-date interpretation and systems so with this in mind I would like actually to point you to a variety of algorithms that we have discussed so far you can find a lot of them on our website maybe more interestingly we have just released yesterday a collection of software that underpins some not all of what I have discussed but quite a lot of what we have discussed so if you would like to see our algorithms and take them elsewhere this is an opportunity to do so and I also like to say that a lot of what you saw here both algorithms but also this demonstrator that Jen and I have built would not have been possible without a collection of brilliant PhD students in my group at UCLA in Oxford at the Turing and in Cambridge and also of course the national disease registration service in public housing thank you very much [Applause] [Applause] I've heard some thing that is dramatic and thought-provoking and we can see what we're on the edge of but we have about 20 minutes left for questions so who would like to kick-off we have two microphones Jessup and Anitra so will please take a microphone please say who you are and your organization they just really have some background when you ask you a question so who's going to kickoff please middle of that side if I can point Richard Alexander at College healthcare you've spoken fantastically and impressively about the advances in technology if I can equally impressively about the advantages and the processes for gathering processing and representing information on a traditional model of effective change I wonder if you'd care to offer an observation on how people and behaviors of the decision-makers are moving such that they can use that advance because I've not heard much about clinicians patients and indeed even how researchers would use this thank you I think this is a very important and brilliant question because like you said as you are providing feedback to the clinician and the patient possibly about different recommendations about actions their own response is going to change so part of the technologies that at least we are building in our lab is we are building reinforcement learning systems that are learning from the clinicians what information they found useful what information they did not find useful and they are teaching through this feedback the computer systems we are building such that they can provide information that's actionable over time but what is maybe even more interesting is an ongoing project that kind of just started together with John and his registration team what we are trying to do is we are building machine learning methods that are aimed at supporting the registration staff such that they can do better and more accurate registration so we develop technology to enforce and enhance the way in which they are interacting with the system so from my point of view again it's a very important area we are learning from the feedbacks patients and clinicians as well as in this case registry staff is giving and the system is evolving on the basis of this feedback I mean I suppose the other point to I'd Richard it would be to say that I mean as we showed in the talk the other thing is the crucial role of transparency in the systems and processes you're trying to introduce you know in medicine I think it's fair to say people are quite conservative and it can be difficult to introduce something new especially see something that is difficult to understand I think you know one of the things we've worked on not just the learning that we can get from individual clinicians but actually can we increase the transparency the the the testability of the systems that we produce down on the front row here just please can you show me one or two hung anybody else who yeah okay Johnny I get you yep Willem oh and Cambridge hematology delightful presentation although to be frank a part of me Ellis modelling I will not follow easily so here comes a question if you're and I think for both of you if you're modeling of the data that are near real-time start giving decision support nurtures to the clinician or the nurse who sees that patient and it would alter the normal care paths what mechanism do you bring in in terms of randomizing approaches to proof like I actually bring benefits versus harm because we can't I assume we all so you must have been thinking about it and I think it's an extremely important question and a difficult one and if the question is about readiness I think for when it's right to introduce something to me it's about parallel running to begin with and to understand whether you are getting better outcomes through the system we've got from the ones that currently exist I don't see that that's significantly different though from introducing a new medication yeah how you introduce a new medication you've got to know other beneficial effects of this medication and there are perfectly good mechanisms for establishing that I think the beauty we have here is that we actually have technologies that might give that answer faster than you know a very strict randomized control you know possible crossover trial or whatever you need to do to show that a new medication is increase the benefit yeah so I agree with you and and you're right what we are part of the reason I'm working on clinical trials and adaptive clinical trial design is exactly with this type of issue in mind so you are very right what is needed to take this technology to the bedside is really a very sort of validation for randomized controlled trials but not necessarily the randomized controlled trials of the past but rather new designs where we can learn using machine learning to which patients would be beneficial in what type of echo system this will be the right environment to introduce that or provide benefits so adaptive clinical trial designs and the work in that area is a beautiful complement in that direction as well Latinos are that means that one of ten patients who walks into a clinic so first clearly as new unit so the question was I'm just repeating the question in case the camera couldn't capture it so the question was as additional information such as general typically information becomes available how to incorporate that so what I expect is both genotypic information but also other type of information information that may be available from wearables and a variety of other devices which may be equally important if not more important so this is why we are building machine learning methods that are quite general that are able to order push of a button generate a new prediction a new forecast but also are able to tell us about the value of information and the value of information for this particular individual is not a moment in time so it may be that for one particular individual we are going to determine that having their genetic information is going to be very important for the risk and disease profile well for another is their lifestyle their sedentary lifestyle and their monitoring of let's say stress that could lead to their impact in the long run so what we are building is technologies that are able to be at scale automatically incorporating new sources of information and being able to tell us the value of this additional information over time and the individualized level so one slide that we kind of went very fast at the very end was saying no longer have these static cohorts where I built a cohort and the cohort is there and I'm asking questions to it we are moving to a new way of thinking about cohorts and data creation where we are learning the for certain impatient acquiring a certain information is extremely valuable and we need rather than spend an enormous amount of money to get that information about everybody we may just be interested in those few patients that will most benefit and use the available amount of money remaining to invest in collecting another source of data so what I see that this joint research can go in the future is also understanding what information would be valuable for whom and built if you like online gathering systems that can can teach us about that but I I heard a slightly different question which was how would we cope when one in ten people have a genome linked to this clinical data set and I think that that's extremely important because I think you know although the success of a hundred thousand and of the biobank is clearly there they're significantly underpowered and actually what you need you know when you've got three times ten to the nine bases and very now at large numbers of significant variants of unknown significance linked to a rich clinical dataset then you can begin to get insight into what those means and the complexity there is the combinatorial issues of different points on a genome that are separated by mega bases can I can I add a comment before I go on to Janice you'll forgive me really from those questions so I'll use my prerogative to it's a comment more than a question but right at the outset I said one of the critical things is the evaluation of treatment plans and one of the things that I've learned tonight is that so having the kind of time series data that is in the register you know I can actually see that happening what I'm not here about is whether there is a measure there which is better than didn't die as soon as that would have done if it had another treatment yes yes so I think yeah yeah this is a very fair question and you know clinical data called a tree is difficult to get and death data is probably always seen as one of the most accurate and therefore we tend to use it but actually we are getting a much much richer data set so let me give you some examples you know time to recurrence and relax yes and repeat tumors we now have links to primary care prescription data which actually once you begin to look at prescription data you can begin to say symptomatology so even if it's a soft assignment say as depression or other treatment modalities we can get very good timed events that are beginning to tell you about real qualities of life and patient information so I think it'd be good to turn those into formal variables to go into the system yeah okay any further janet next well I think so it's Janet Allen from cystic fibrosis trust I think will and was asked the question I really wanted to ask which was having you showed examples were treating breast cancer you would suggest chemotherapy alone as opposed to radiotherapy chemotherapy at the end of the day medicine is very highly regulated as well as very conservative profession and you're not going to have to go back and do randomized controlled trials of chemotherapy versus chemotherapy and radiotherapy in order to prove to provide the evidence that your prediction is correct but I think bullying was asking a very similar question from the point of view of how do you take this forward from where we are I I think it is a you know I I think this is the problem the medicine has to face I'm not trying to dodge your question but I think you know where I started was that we're working in the dark and that we are making guesses based on rann true randomized control trials into how we treat people now the problem randomized control trials doesn't mean error says you know the small numbers the perfect population that's used in them does not so from population to individual is not working and you know anyone who's practiced medicine knows that you know a large number of patients don't get better when you actually intervene with them yeah and we're working in the dark and I think what is going to happen in medicine is we will see a fundamental shift to doing things algorithmically now the question is how do we increase that trust and ability to do that but actually you won't be able to deliver that when someone's got a genome attached to a complex clinical phenotype without the sort of systems we've shown here and our vision is that's what we're trying to build degree one a few more questions in the middle please can you pass the microphone in that direction it's probably the quickest way to do it I'm Kevin Flanagan a software developer and I've got a question really for you about robustness of the model when it gets to implementation because in the real world people miss data points or miss key and you know knowing these algorithms as well as I do one you know one of them dominates and that's the one that's miss Kay you've got a problem have you thought about that yeah so I worry about that a lot and I think that this is a very important question so what happens when we have for instance also missing data what happened when we have noisy data possibly erroneous data so actually my group has developed quite a lot of algorithms to deal with missing data inclusively data that's missing not at random so what would happen if there is an information that is missing but that may be informative so with missingness we have done quite a lot and trying to understand what's the value and the value of information the issue of robustness of algorithms is also we have looked quite a lot at it what can we say about algorithms are going through business how they stay robust with in subpopulations with the rest a robust if we take them to another cohort that may be slightly different all of that we have done but what we have not done and and was also implicit in your question was what happens if the data that we put in has quite a lot of noise so it's not missing but there's a lot of noise that's an important area we are working a lot about and we are thinking together with German law that both to be able to do increase the data curation through for instance in technologies that we are developing to help gather more accurate information but also to learn using these machine learning methods what data we can trust for a particular individual and how that impacts the predictions that we are making so again so the question was about spurious effects and maybe these algorithms may provide or or they may discover something that's kind of obvious in retrospect or or not obviously that respect so that's why as Jen was mentioning before we focused a lot of interpretable models both for the longitudinal trajectories but also try to interrogate what variables are important why these variables are important in the individual why the predictions for this individual was different than the population try to understand variable importance and individualized level for this particular prediction as well as over time so we are building a suite of algorithms to kind of interrogate this black box models but what the next frontier is is to bring these models and this understanding to clinicians and to services like the ones gem are building such that we can jointly debug them and understand whether certain models are indeed more robust whether a certain way of selecting variables that to put in a modeler is more robust and what is the sensitivity associated with these variables and the predictions now they might say I'm not sure this is a new problem you know you've got is the latest measuring equipment and ways of doing it you know mitten and and the entire life is full of you know inappropriate associations you know spurious correlations I think you know we've got a new matter that are discovering them that doesn't mean we should abandon common sense clinical practice you know gut feel even we will find them they'll be there there are methods we can put in to try and minimize them we look forward to finding them with you now I'm determined that we end round about eight o'clock because that's really promised we would and so I'm going to say that's the end of questions and it and it leaves me just to say to three things to thank our two speakers I want to start by making two points there is a kind of add-on if I'm allowed to do that one of the things is you know I learnt quite a lot about interpret interpret ability tonight and there are many other fields that I suspect me the kind of expertise that Murray was talking about with using non interpretive or algorithms the second point that I wanted to make is and I learned this from mihaela over the last couple of years what we're hearing about is assess of what they're actually generic algorithms and one of the things we're doing for example in the journal Institute is looking at the possibility of converting this model phone call it you know whole model system to deal with not patients but offenders in the criminal justice system and if you start thinking about through there are all sorts of other possibilities in the fields where this work has no history is not started and I think the potential is tremendous so I think what we're hearing you know has wider applications and is very exciting but I then want to say things about this evening I and I'm sure all of us have learned a tremendous amount about the Cancer Registry the richness of the time series data and the professionalism of Jim and his colleagues which is going to make actually this possible and we've heard from the hailer the ways of extended methods to actually deal with the problems you know she still is very elegantly through where we now are which why which in really meant two or three years ago and you know and what's been achieved in the last couple of years to really move this field forward and I think the beauty of German mihaela doing this is a joint venture is it also shows the power of kelabra and I think that is incredibly important so we can all see now you know what can be achieved and what will be achieved and I think we're actually seeing the future of Medicine so very grateful to you both tremendously no thank you [Applause]

Leave a Reply

Your email address will not be published. Required fields are marked *