Using Deep Learning to Transform Radiology Practice with Kevin Lyman, Chief Operating Officer and Lead Scientist at Enlitic

Thanks for tuning in to the Outcomes Rocket podcast where we chat with today’s most successful and inspiring health leaders. I want to personally invite you to our first inaugural Healthcare Thinkathon. It’s a conference that the Outcomes Rocket and the IU Center for Health Innovation and Implementation Sciences has teamed up on. We’re going to put together silo crushing practices just like we do here on the podcast except it’s going to be live with inspiring keynotes and panelists. To set the tone, we’re conducting a meeting where you can be part of drafting the blueprint for the future of healthcare. That’s right. You could be a founding member of this group of talented industry and practitioner leaders. Join me and 200 other inspiring health leaders for the first Inaugural Healthcare Thinkathon. It’s an event that you’re not going to want to miss. And since there’s only 200 tickets available you’re going to want to act soon. So how do you learn more? Just go to outcomesrocket.health/conference. For more details on how to attend that’s outcomesrocket.health/conference and you’ll be able to get all the info that you need on this amazing healthcare thinkathon. That’s outcomesrocket.health/conference.

: Welcome back once again to the outcomes rocket podcast where we chat with today’s most successful and inspiring health leaders. Today I have an amazing guest for you. Someone that’s had a contribution in healthcare in a lot of ways from the bench to the frontlines. His name is Kevin Lyman. He’s the chief operating officer and lead scientist at Enlitic. Enlitic is a medical Deep Learning Company. And twice named one of MIT technologies reviews 50 smartest companies where he applies deep learning to the early detection of cancer. Kevin is also the founder of the inventors guild a team of over 40 students from 10 of the world’s top universities who earn course credit and money consulting for startups. Competitive at heart, Kevin is a former professional Halo 2 player and was once the world’s highest ranking Warlock in World of Warcraft. Not an easy thing to do ladies and gentlemen. He has since focused his energy elsewhere winning over a million dollars in the words from hackathon and business plans to competitions. And now he’s with one of the most forward thinking health companies, Enlitic. So it’s a pleasure to welcome you on the podcast Kevin.

: Pleasure to be here and thank you for having me.

: Absolutely so anything that I’ll leave out there in your intro that maybe you want the listeners to know about you?

: No, I think you covered my background pretty well. It’s been a long journey to end up in healthcare but I’m pretty excited to be in the space now.

: Hey man that’s really great. And you know I’m just curious why health care. Like how did you end up here?

: Well I come from a family unfortunately with a lot of medical disability. In particular I grew up caring from my mother who loved bed down for multiple sclerosis and so that had me sort of growing up in that environment where I was forced to recognize the importance of patient care and that exposed me a lot to being in and out of hospitals and needing to care for in different ways just coming very close to that type of problem. And throughout the remainder of my childhood unfortunately a number of my other family members suffered from many different types of cancer and other rare abnormalities. And so it’s always just been something on my mind. But even outside of that the most important thing to me has always been use cool technology to solve important problems. And in the quest to do that, I had started a couple of companies one of the machine learning space on the medical device space and running both of those concurrently. I’ve been introduced to Jeremy Howard the original founder here at Enlitic and he just really got me inloved with this idea of the intersection of those two things of machine learning and medicine and really just made me realize that even being an engineer in today’s world, I have an opportunity to be a doctor to then ever since I’ve just been really in love with that idea.

: That’s super cool. And what a great story of how you ended up doing more work within medicine and and folks Kevin’s been a speaker at TED MED. He’s also been a participant of the hive. So he’s he’s doing some really neat things here in healthcare. So thanks for sharing your story of how you ended up here. So sorry to hear about your mom and sort of the hardships and it sounds like these are the things that really kind of formed your basis your empathy which I think is something that is missing in many health tech companies. So really great to hear that that you do have that I couldn’t possibly agree with that more and I appreciate you saying that.

: I think that’s probably been the biggest learning experience for me being in healthcare that unlike any other industry I’ve been in, empathy is more important here than anywhere else. And every decision really needs to be made with the mind set of how will this impact patient down the line.

: Man I couldn’t agree with you more. So tell me Kevin, what do you think today is a hot topic that needs to be on every the leaders agenda and how is Enlitic approaching that?

: Well I think I’m obviously a little biased here and I’d say that intelligence and I think that that’s probably becoming increasingly more and more unavoidable from pretty much any health care professionals perspective that AI is attending Ubiquiti and it’s getting to the point where these promises really are coming into fruition and very, very quickly. But given that that’s something that is really on its way in I would say the thing that people really need to be worried about right now is how to prepare the infrastructure for the largest for AI but for all of these other technical innovations that are right down the road if we don’t set up our systems now to welcome these new tools in, now we’re going to have to deal with that in the future. And even though we don’t know exactly what all of these solutions look like today I think we do have a reasonable idea of what needs to be done to get our infrastructure into place to actually implement some of those solutions. And so as far as actionable things go today I think that’s really what should be on everybody’s minds just infrastructure and planning for that inevitable upcoming future.

: Kevin, I think that’s such a great point. In fact I’m putting together health care meetings starting in September where our theme is innovation is implementation in healthcare. And you got to be able to implement and you’ve got to do it well for the sake of the listeners, Kevin, I love if you could just level set everybody on Enlitic. I don’t want to assume that everybody knows what you guys do who is Enlitic, what problem you guys solve, who do you solve it for?

: We’re a team of about 16 data scientists and fulltime radiologists out of San Francisco along with a team of about 60 plus part time radiologists that work with us and we develop clinical applications of artificial intelligence. We do this sort of all across the spectrum of digital diagnostics but our primary focus is within radiology and so we train neural networks and other forms of machine learning models to interpret medical data primarily medical images and text in order to assist radiologists and other diagnosticians with the clinical diagnostic process and that can mean a lot of things. But in general we try to enable them to perform diagnoses faster more accurately and in many cases with insights that they otherwise couldn’t. And a lot of our focus up to this point has been BORGIALLI in chest imaging. So building solutions to read things like just x-rays and chest CDs but we’re increasingly expanding that aiming for whole body coverage over the next three or four years hoping to have models that regardless of where the images and what type of study it is ideally have something that can work with the doctor to enable them in that diagnosis. And I think there’s a lot of many very different types of solutions. We’ve implemented to this effect. But for the most part they’re usually focused around trying to spot things within the images and help radiologists understand what those are and how to treat them.

: Outstanding. Now thanks for that level setting listeners check out their website. Go to www.enlitic.com that’s E N L I T I C, dot com and you’ll see some of their applications. Some of the news, some of the updates, their mission. I think it’s really interesting the way that they’re using deep learning as a tool to augment the work of radiologists and it’s pretty inspiring stuff that you guys are up to there Kevin. Tell me can you give us an example of how you and your organization have created results or improved outcomes by doing things differently?

: Yeah I think that this starts to play a little bit back into that whole empathy thing but one of the big insights that we’ve had in working on these problems is that AI is very much a user experience problem that I think a lot of people like to look at AI development as being hardcore software development. I think when you are very close to the field it’s very easy to assume that but it’s an incomprehensible amount of very intimidating math, a lot of software that might require a lot of specialization to understand. But what I think the helpful way to look at it is that the art of building AI is really building black boxes. It’s building a system that is really good at taking a given input and extracting a desired output from that. And so once you’ve built that black box, the remainder of the problem really becomes what do what you want to go into that black box and what should subsequently come out of it and to train that black box requires lots and lots of examples of that input output pair and so very quickly it becomes a problem of working with people more so than crafting good software. So we’ve taken this approach then of building our own network of radiologists and building our own tools that these radiologists can use in order begin to label historic data to generate this input output pairs that we can then trade models on. Then we’ve had to go this route of building all of this in-house because while tools exist that can be used for these purposes they don’t really enable you to be very expressive. And another very important insight that’s come out of this is that you can collect all the data in the world but ultimately it’s meaningless if you didn’t ask the right question when you were collecting that data then I like to use an example of that in the interpretation of just expertise where we find that a lot of people these days are trying to train models to identify things like tuberculosis in a chest X-ray. But unfortunately people don’t diagnose tuberculosis with just the x rays and so it’s a little jumping the gun to try to shoot right for a model doing the same that what people look for as far as TB goes in a chest x ray or signs they look for consolidation of calcified lung nodules, plural of fusions, things that are indicative that the patient might have TB, but in reality you need more information than just what is present in that image. I would also need to know about your clinical history. We need to know what country you’re in. I would need to know if you travelled somewhere recently where TB is very frequent. If you’ve been coughing and unfortunately the X-ray alone doesn’t have all of that information in it. So if the question you ask your radiologists when they’re labeling your training data is does this patient have tuberculosis will still try to answer it for you but will be answering a flawed question. Unless the data you’re gathering is flawed. And what we found is a must we’re building these tools. We can’t structure it the right way to ask the correct questions. And unfortunately you don’t realize you’re asking the wrong question until very far down the line. And so that’s why we wanted to make sure upfront that we’re building with this kind of concept in mind.

: What a great insight you’ve provided there Kevin. And oftentimes you don’t realize this like you said until much later in the process. Can you share with us a time when you had a setback and what you’ve done and what you learned from that?

: Yes actually the example I just gave is a good example of exactly that. The very first thing that we built was a fracture detector because..

: Ah, OK.

: One of the most common types of things that people come to the doctor with as far as a radiologist goes are fractures and then often in the extremities like a wrist fracture for example. So we built a model that could ingested an x-ray of the wrist towards the arm and would automatically circle a fracture in that scan if it found one. The intent being that when the radiologist brings the scan up it’s a lot easier for them to find the fracture if a model already did it for them.

: Right.

: And when we collected the data for this the way that we tackle it, this was before we had all these systems we’ve built ourselves in place. We looked at what sort of image reading tools for radiologists exist today that are open source that we could mess around with and we chose one particular and set up a bunch of mac books for a radiologists to use. But all we did was load those x-rays of historic x-rays from our partners. And we asked the radiologists to circle the fractures and unfortunately that is not a very specific set of guidelines to give them. And so the data we got back even though you and I as non radiologists circled the fracture might seem pretty straightforward to them it could mean a lot offense. What do I do about the non wrist fractures. So the fractures that exist in the elbow or the fingers were somewhere else. Now what do I do about sesamoid bones which look like fractures and are often reported like them but are not fractures how tightly do I contour anything that I circle that was a big problem we and where when we trained the model the first time we found that it was getting all sorts of false positives and false negatives. So the first thing we did was we went back and we looked at the training data collected by each of the radiologists that participated in this exercise and we found that one person was just completely off the mark. They missed all sorts of fractures. And so we completely wiped them from the training data and we retrained the model and we find that that drastically improved results. And once we got rid of the bad actor things improved but we still had a big problem which was that the model in addition of getting the fractures correctly was circling all sorts of little artifacts that were not fractures they were just meaningless little blips. And again looking at the training data we realized it was because one of the radiologists who prepared training data did not know how tightly to contour them because we didn’t tell them how tightly contour it. So he circled every fracture as tightly as humanly possible but a fracture just looks like a line. It looks like if you were to pluck a hair and put on a white piece of paper it’s just that very thin line. If you zoom in on it enough anything looks like that any digital image when you zoom in enough looks like just a lot. And so having the model placed attention just that tightly on the fracture caused it to lose sight of the fact that the context around the fracture is what makes it a fracture. It’s not just a line anywhere. It has to be aligned through otherwise healthy tissue. So what we did was we went back and we modified that training data to programmatically expands the size of each of these regions of interest they drew on the pictures by 30 percent and then retrained the model. And when we did that, we got amazing results. But the big thing that it really ingrained in us was this understanding that you have to be extremely specific when you’re asking these questions where people are free to make up the rules on their own right. So now every time we launch one of these tasks it usually comes with about a hundred pages of these guidelines and several rounds of tests that we make them go through to begin with.

: Wow, what a lesson learned and sounds like at this point for the chest imaging you guys have set some parameters very specific guidelines to ensure that you guys get more success upfront.

: Yup absolutely. And that’s been where a lot of the difficulty comes in.

: My goodness. And it’s a lot of work. You guys have to do all this stuff up front but at the very least you get what you need and and when you don’t know what you’re looking for you don’t know what you’re looking for right now. Now you do.

: Yeah. No, actually it’s hard enough that most of the time our task is finding any number of nondescript needles in a haystack. So we’re learning more and more specifically about the problem.

: It’s a good way to put it. Kevin what would you say to date is one of your proudest medical leadership experiences?

: You know I think it’s hard to answer that specifically limited to medical leadership but I think it’s been incredibly rewarding to see as the company progresses just that are the way that we vocalize our meticulousness about this data because of how ultimately this is how we ensure the best patient outcomes to see how well received that’s been in this evolving market. There has been great and very rewarding because early on I think there were a lot of deceptive actors in this space that made a lot of very bold claims about what this technology could do and very quickly people realized that the promise is not there yet and it soured them toward the market and it’s been very rewarding to see that with us being up there being very transparent about this being the voice of this is what we really can and cannot do in this space. How well that’s been received I think that that’s I say very rewarding because being in healthcare transparency is everything. And even though this is the intersection of technology and healthcare I think that’s more true now than ever. And so it’s amazing to see that other people agree with that message. And that that’s something that’s leading to success. But outside of that to sort of nail to some concrete moments, I think one thing that was very rewarding for us was last year doing a blind test. And this was something that I alluded to when speaking at TEDMED that we did a blind test mobile group radiologists that found that our lung cancer models weren’t just picking up lung cancer but they were often discovering malignancies up to 18 months earlier than human radiologists were that this emergent insight came out of this approach to knowledge discovery and data integrity. I think really illustrates the true power of these types of solutions.

: Wow that’s really interesting 18 months prior?

: Yeah.

: That’s amazing. Well congratulations Kevin. I mean no doubt that it’s tough right. Because when you’re in the market and you see the potential in a technology what it could do and then all of a sudden the bad apples will column come in and sort of muddy up the water. You guys have stayed consistent and you’ve stayed true to making this data as as clean as possible, working with the best parameters to get the results that you guys are looking for and you’ve gotten some wins so congratulations on that. And these results I mean 18 months beforehand. I mean that’s huge.

: Yeah and this is those are the main reasons that we pick lung cancer is an early thing you’re working on. There were some abnormalities that it doesn’t matter how early you pick it up. Unfortunately there’s nothing we can do. But with lung cancer, the early you pick it up the more likely the patient is to survive. Then I think 18 months is quite significant. And so we really hope to see that be something that it’s clinical practice if not this year or the next.

: That’s wonderful. And the other thing that you said Kevin that really struck a chord with me is the other factors. Right. You can’t just rely on the image itself what’s the context of this image? Today, I think there’s a larger influence of the Social Determinants of Health and the influence that they have on specific diagnoses. And you talked about hey was this patient traveling or where did they live or what did they do. And these these things are now also included in the models that you guys are putting together?

: Sometimes those factors are put into place sometimes they’re not. It really depends on the specific problem that’s being solved. But in the case of trying to build a general system for the diagnosis of chest x-rays just to limit the problem the one specific study time. I think it’s critical to keep in mind that it’s not enough. Just look at the picture. And if that’s all you have then the answer you provide should take into account that that’s the only information you have. In clinical practice. The hope is to use not just the report the image itself but any report that might exist about it. Any patient history that might exist part of the reason that we have that 18 month early detection is because of our use of longitudinal data. So the patients whose data train that model had several years of scans that the model has learned from and as well as a lot of demographic patient information and biopsy results and all sorts of other clinical data that has enabled us to extract those emergent insights then more and more so you’ll see that become the common way that these systems are implemented where you’ll have multiple models in tandem reading lots of different types of data and then they’ll marry that output in the end in order to give you a much better representation of what we believe is going on.

: Fascinating folks again just a reminder If you’ve tuned into this podcast a little bit later I’ve got Kevin Lyman here chief operating officer at analytic you could check them out at www.enlitic.com. They’re doing some pretty amazing things. They’re at the forefront of AI and in radiology. Check them out. Press rewind on this podcast because conversations have been really good. So tell us a little bit about an exciting project that you’ve been working on today?

: Well all of it is very exciting. By the way I think you know one thing that is going down a wall unfortunately my way of boiling it down will be to expand that a little bit further back. I think one thing that’s got us all extremely excited right now is we’ve had a lot of innovations around how to scale this process horizontally across the vast majority of digital diagnostics, as I said it’s helpful to think of these systems as a black box that we can build a couple of specialized black boxes. But at that point the problem becomes around tuning it to all these different data sets that we collect so that it can work to a lot of different types of problems and that’s really what we’re focused on right now. How do we take something that does fracture detection and arrest or or lung cancer detection in the chest and now translate that to look for anything anywhere in the human body. And that’s something that we’re currently fundraising for right now. And as we close that round we’ll be moving towards actually starting to act on that scale and I think that’s really where we’re going to see some of the biggest patient impact down the line because up to this point each of these solutions each type of study each specific problem you tackle has been a massive lead time in the development of those systems. Look at how long it’s taken us to get where we are today or how long it’s taken our competitors that many of whom are also doing good work to get where they are. But the exciting thing I think is as far as what we’ve recently come across is now greatly reducing that cycle to be pumping these systems out with very high quality for many different types of problems on a very short turnaround time.

: Truly exciting and it’s awesome that you guys are aiming toward this broader spread of these solutions across different applications. It’s pretty exciting man. Congratulations on that.

: Thank you.

: So Kevin time does fly my friend. We could talk about this stuff all day but we do have limited time here on the podcast. We may have to do a part two but for now we’re getting into this part of the podcast where we walk through set up a syllabus for the listeners. Since you mentioned earlier today sort of the importance of setting up the appropriate infrastructure for these types of innovations I love to take the focus of this course that we’re going to put together for the listeners and the lightning round around that structure implementation idea. If you’re good with it.

: Yup.

: Awesome. So we’re going to call this the 101 of Kevin Lyman on setting up the infrastructure for innovation. And so as we go through this lightning round I’ve got four questions for you around this focus. And then we’ll finish that up with the book that you recommend to the listeners and a podcast. You ready?

: Perfect.

: Awesome. So what’s the best way to improve healthcare outcomes?

: I think within the context of setting up infrastructure for any kind of administrator or leadership at the hospital I think it’s important to re-evaluate the systems that are in place today. And I think we find from a lot of the partners we work with that they’re still using systems from 10 plus years ago. And I know that all of these tools are very expensive to put in place and there’s a lot of risk involved. But technology has progressed quite a bit in the past decade and I think it’s really time to start updating some these tools to be more receptive to the incoming solutions from A.I. and other improvements and software that ultimately are going to lead to much better patient outcomes. But only if we set them up for that right now. And I would extend that by saying just a little bit but nothing is more important than patient safety. Then of course what stems from that is patient privacy. But anonymization is quite good today and so is the siloing of patient data. And while it might be very scary to give up patient data in this way I think the appropriate systems are in place these days that the risk is greatly reduced. And I think it’s worth the risk, it’s justified by the benefit that comes from sharing this information with the research community that can do amazing things with it.

: Outstanding. What’s the biggest mistake or pitfall to avoid?

: Surely within this context of infrastructure I think getting to married to any one particular system can lead to big problems. I think we found with a lot of our partners that they put one system in place and it introduces problems. And rather than fixing the underlying problem they put more and more bandaids in place and so ultimately 10 years down the line when it is time to update that infrastructure. It’s a massive amount of work in undoing all of these individual solutions that were put in place. If you come across a problem you solve the underlying problem.

: Great message. How do you stay relevant as an organization despite constant change?

: I think in our case we thrive on that change is where research organization but for a lot of these other groups that are sort of stuck in ways where it’s a lot harder to adopt this change. I think it’s important to listen to the research community and listen to evolving methods of how one measures risk in the field. I think the FDA is increasingly catching on to that and is growing more and more away from lots and lots of expensive clinical trials upfront and more and more toward what’s pulling in the fields after doing a little bit of checking and then monitor it in the field to see how it does. And I think that this is a really powerful philosophy that in software development is pretty much what we call agile. But I think that hospitals in particular need to adopt this mentality a little bit more and understand that while there is risk involved in taking some of these leaps, you can minimize that and you’ll get a lot better data by taking the leap and seeing how it goes. As long as you get the appropriate fixes in place.

: Beautiful what’s one area of focus that should drive everything in a health organization?

: Patient outcomes that should be the number one thing that drives every decision. And I know that a lot of times financial decisions are hard ones that need to be made in order to optimize for patient outcomes in the grand scheme of things. But ultimately that’s the only reason that money should be the focus in mind. For all others it should always come back to patient outcomes and everything should always derive from that.

: What a great syllabus here listeners. This is again directed toward the implementation of infrastructure to support new technologies. Kevin Lyman here from Enlitic. So Kevin what book would you recommend for the listeners?

: My favorite book is the 80/20 principle. I think I’ve learned a lot both from the book and from the principal in and of itself that in many cases you can get 80 percent of the value with 20 percent of the work and that is in no way me condoning only doing 20 percent of the work in certain circumstances. But when you’re in a very multidisciplinary field, the way that we are it’s critically important to be very well educated not across just artificial intelligence and computer science and software engineering but also clinical science and medicine. Then the reality and gravity of the practice that you’re trying to build solutions for and we’re in a multi disciplinary field like that it’s a very helpful mindset to understand that I will never be the best doctor in the world but if I’m smart about whom I spend my time with the people that are the best doctors in the world then I can learn enough of what they know to do good work in my field.

: What a great message man.

: I just want to make sure I also give a quick shout out for the tech intern blueprint. It’s a book that was just written by two of my colleagues from the inventors guild or any software students or engineers out there that are graduating college and looking for a good tech internship. They’ve put together an incredible book full of Amazing Tips and tricks on how to optimize your chances of getting your dream job or your dream internship. So check in turn blue print. Check it out on Amazon.

: That’s awesome. Hey thanks for the recommendations Cavin listeners. Don’t worry about writing any of that down. I know we’ve been talking about a lot of great things. Just go to outcomesrocket.health/enlitic that’s E N L I T I C. outcomesrocket.health/enlitic. You’re going to be able to find a transcript. All of the show notes and links to all the resources that our friend Kevin Lyman here has been sharing with us. This has been a ton of fun Kevin and I’d love if you could just share a closing thought. And then the best place for the listeners could follow up with or follow you.

: Yeah absolutely. I think I always like to close these things just by telling people that they should be very excited about the future of health care because even though it’s a field that from a technology standpoint hasn’t seen a whole lot of rapid evolution, now is the time that really we’re seeing more work done than ever. Well a lot of us would probably consider to be science fiction is actually creeping up on us a lot quicker than we could anticipate but in very good ways. And so it’s something that I find very exciting and I hope to leave everybody very excited about this as well.

: Thank you so much for that and Kevin what would you say the best place for the listeners to get in touch with or follow you as?

: You can follow me on Twitter @ktlyman, or you could always e-mail me kevin@enlitic.com. I’m happy to answer any questions you have about clinical AI or anything of the sort.

: Outstanding. Kevin this has been a blast really just want to acknowledge and congratulate you and your team for the hard work that you’re doing and the advances that you guys are making. So just want to say thanks again for spending time with us. Looking forward to staying in touch.

: Fantastic. Thank you so much for organizing it was really great chatting with you.

Thanks tuning in to the outcomes rocket podcast. If you want the show notes, inspiration, transcripts, and everything that we talked about on this episode just go to outcomesrocket.health. And again don’t forget to check out the amazing healthcare thinkathon where we could get together to form the blueprint for the future of healthcare. You can find more information on that and how to get involved in our theme which is implementation is innovation. Just go to outcomesrocket.health/conference that’s outcomesrocket.health/conference and be one of the 200 that will participate. Looking forward to seeing you there.

Automatically convert audio to text with Sonix