Eisenberg: Welcome to this second episode in our mini podcast series on building Dyna AI. My name is Kate Eisenberg. I'm the senior medical director for Dyna AI. I’m a family physician, epidemiologist and informaticist, and over the past couple of years, we have worked across disciplines, across roles to build a clinical decision support platform - Dyna AI - that's grounded in evidence and shaped by real world clinical expertise. This is a mini series to reflect on that work, what we've learned, kind of celebrate the people who have gotten us here in what has been a very fast paced ride. So, for today's episode, I'm joined by Al Stevens and Ben Hollis, who are both really key leaders in this effort. And I'm going to let them introduce themselves, so Al I'm going to start with you.

Stevens: I'm Al Stevens. I've led the project from the beginning overall. And, you know, I didn't invent the idea. It came out of our technical staff. I think my role was actually to recognize that there was this new technology that we could use to answer clinical questions to make our information more readily available to our end users. My background is in old school artificial intelligence. I many years ago ran a group that approached these problems using symbolic AI. I happened to have had some experience during that with the nascent world of generative AI and that helped me recognize that this was going to be a technology that we could effectively use. I'll save more for later, but that's kind of the big picture for where I came into the project.

Eisenberg: Fantastic. So you had that long view of, okay, this is something to pay attention to just from the beginning. 

Stevens: I think so. Not without some skepticism. I mean, to be very honest, in the symbolic AI world that I came out of, there was a lot of skepticism over generative AI. I mean, I have a professor friend who is more classically oriented, and he was sending me messages, telling me why this was never going to work. So, I think, fortunately, we could see enough examples in our own world that showed it was going to work that we kept pushing it. And I think that's one of the reasons we're here today is because collectively, we were able to push this along.

Eisenberg: I love that. And, Ben, can you introduce yourself, please?

Hollis: Sure. My name is Ben Hollis. I'm on the product management side here for Dyna AI. My role involves a lot of customer facing elements - working with the market, ensuring that feedback makes it to the technology and clinical teams and that we keep driving forward for customer satisfaction and customer delight with Dyna AI’s capabilities. My background is pretty varied. I actually have some background in editorial, working in journals as well as marketing, technology and most recently some education training and work in informatics, which led me actually to the Dyna AI team. I was lucky to have my first week here correspond with the innovation that Al mentioned that our technology team shared though. Kind of feel super, super lucky to have gotten in on the ground floor with Dyna AI and be along for the ride as this really strong team pulls things together and pushes great value out to the market with this solution.

Eisenberg: Love that. So, I, you know, you both introduced yourselves and you come from different backgrounds and obviously that shapes your approach to applying an innovative new technology in practice. So I wonder if you could share a little bit more about how that background coming from a really deep, you know, immersion in AI, going back well before generative AI came on the scene to now, you know, how does that shape your thinking and your future thinking?

Stevens: Well, as I said, I brought to this a very skeptical attitude. I mean, I was one of the people early on who was questioning whether we could do this at all, and even at times wondering whether we could, but part of my background is the scientific training. So, you know, what matters is the data. So we just started trying some things. We tried some simple models. We tried taking chunks of our content and asking and throwing that into a model and asking questions. And as we saw, as I saw and others saw, what was actually happening in the quality of the answers we were getting, the ability to summarize, the ability to even do certain kinds of inferences, it became clear that this was a direction we should be pursuing. And we also, as part of that process, were able to look at, you know, the areas people were telling us where things might not work. And we were figuring out ways to work through those and actually make them work. Generative AI is a really complex tool. Like a lot of complex tools, you have to understand it and understand it in detail, understand what it can do, what it can't do. And that's been a key part of this process of understanding where it has value, where you have to be careful and where it simply doesn't work.

Eisenberg: That makes sense. And so you were really bringing that very high level sense of how generative AI might fit into the overall landscape as a tool.

Stevens: I was trying to do that, yes, yes, that was one of my roles. Yes. 

Eisenberg: Fantastic. And Ben, over to you. You know, you described a really interesting, varied background. And I will say being a part of this team has brought variety, among many other delights and challenges. And so I would imagine that that background has served you well. I'd be interested in hearing more about your thoughts.

Hollis: And I think, you know, I shared a lot of Al's skepticism about where we might take this and what the capabilities were. Most of my more technical background is in discoverability, search, content architecture, information architecture, that sort of thing. And so as generative AI sort of started to come on to the scene for all of us, I think there was a healthy level of skepticism about its capabilities. But I think, you know, one of the things that this team, you know, with its varied backgrounds, its approaches, has always maintained consistency around is, in an innovative spirit along with a willingness to experiment, to fail, to try and to really focus on iterative development. So I think, you know, the composition of this team, the very background leads to a very sort of iterative, technology-driven approach. And so I think that skepticism for me sort of started to evaporate when we began very early in the process taking our ideas and putting them in front of potential customers, where our beta customers or we called them innovation partners and started to really bring this early form of Dyna AI out to the market. And even though it wasn't perfect, and still very much is today a work in progress, the response from clinicians who were seeing it, you know, internally at first and then externally was hugely, hugely positive. And I think that feedback really helped drive us on from a motivation standpoint, but also reinforce the iterative sort of experimental approach to continuing to develop Dyna AI that we have today. So I think that's been really one of the keys to success is fast to market but then also fast to gather and respond to clinical feedback from customers as well as developing a robust sort of clinical quality program to help support our level of faith in the product as it comes into the market.

Eisenberg: That makes sense. And I remember that. I remember the way that we kept moving forward early on was take a step, iterate, take another step. And it just kept working so well that we kept going, and we started seeing that value.

Stevens: There were some steps where we had to go backwards though. I mean, one of the things we did that I think is very important in this process is we were willing to fail. We worked to fail fast, to use that term, and when we failed fast, we didn't give up. We stepped back and said, what did we learn from that? And we were able to use that to move forward.

Eisenberg: I think that makes so much sense, that willingness to fail, right? So early on with this technology, you know, you had to know there was a risk of failure just to even take a step forward because it was so new. Ben, how about you? What is something that's surprised you in terms of the challenges and risks in, building this tool?

Hollis: I think the rapid pace with which we as a society, like the market we have in particular, but all of us have sort of started to incorporate AI into our lives. I think that's probably the biggest, most surprising thing about this whole process is just how willing even the strongest skeptics we had, clinical skeptics we had within, you know, our organization came around to the, you know, sort of potential of Dyna AI, and in many ways have become some of its biggest boosters. Right? There was a lot of fear that, you know, and fear which is justified about, you know, the potential for hallucinations, the potential for misinformation. But I think, you know, the strategies we've taken as an organization to try and minimize those risks, to try and really focus on robust clinical quality, have helped us, you know, transition folks into being strong supporters of generative AI in the health care information space. And so I think that embrace of the medical community of AI’s potential is probably one of the things that I found most surprising in a really positive way. I would also say one thing with Dyna AI that I do miss a little bit was in the early days we had, Dyna AI would still be able to answer as a pirate, and so would answer medical questions in a pirate voice. And I maintained from a product management perspective, we should keep that in there as that. There’s a tiny part of my heart that does miss that. But, I think you could still delight a few customers with a, you know, pirate AI assistant

Eisenberg: I do keep those responses, they’re my favorite. Al, there's always this potential tension between speed of development and clinical quality. How would you say you and our team have navigated that tension?

Stevens: I think I'll go back to my point that in some cases, we actually failed. And, you know, we went, we swung the pendulum too far one way or the other. But, you know, for example, when we started, we were so worried about clinical quality that the first version of Dyna AI actually didn't answer directly any question. The answers went into a database, and we asked the clinicians to review them. And once the questions had been reviewed, they went back and became available as answers. That was an extremely conservative approach and we learned that that was just never going to scale and was never going to work. So then we swung the other way and probably released, you know, were producing answers that were less than reliable. I'll leave it at that. But what we were able to do is kind of thread our way through that and provide a balance between the technical side and the clinical side. I want to say that an absolute key to Dyna AI’s development is that the clinicians and the developers worked hand in hand. We met and continue to meet every day. And both sides from the technical side are learning to think somewhat more clinically, and the clinicians are learning to think somewhat more technically. And that synergy is what has got us to where we are at this point.

Eisenberg: Absolutely. Ben, any additional thoughts in this area? 

Hollis: I would just wholeheartedly echo what Al had said about the partnership between the clinical side and the technology side. And, you know, I think there's been a lot of growth on each in terms of the, you know, technology team thinking in a more clinical nature and that the clinical side thinking in a more technology forward way, when and where you and the other clinicians on the team really understand not just how the AI is synthesizing our information to bring a response, but also how the structure of our content, how we write our content, how we create it, how we, you know, grade it all feeds into how well Dyna AI can perform. And I think that's a real asset for us as an organization is the knowledge from, you know, the very ground level of creating medical information, you know, synthesized from primary research into something that AI can use has suffused our organization as a whole through this learning. And the other thing I would say is that it also took a great deal of trust, and I think the team has done a great job building trust between technology and clinical to open those lines of communication and allow effective partnership that I think is really challenging in a lot of organizations that are more siloed, but has been a real benefit to us here in developing Dyna AI.

Eisenberg: I think that that partnership is key. I would absolutely agree. And I think, you know, when people ask me about those first 6 to 12 months of this team, I, you know, I talk about culture building and trust building and it feels ambiguous, but it's really the bedrock that, you know, everything else is built on. 

Hollis: And I think you and all the other clinicians on the team have done, you know, such a tremendous job with, I would say building a grading system essentially that allows communication. You know, in the early days, it was much more difficult for us to convey to the development teams, hey, here's what's actually wrong.

00:15:05:10 - 00:15:24:13
Unknown
But I think now there's been such practice from a clinical and analysis perspective and so much structure built that we have a robust, you know, not just the tools in place, but a way of communicating that says, okay, this is the category of problem that I am seeing in this response. Here is how I think it might be fixable. What do you think from a technology standpoint? And I think that sounds very simple to say that now, but from the process, it's not a simple process to get to that point of awareness.

Eisenberg: Right. It's that whole idea of, you know, ‘I would have written you a shorter letter, but I didn't have time’ kind of thing.

Hollis: Exactly. 

Stevens: Yes, yes.

Eisenberg: We're going to get a little cheesy now and I'm going to ask each of you what's something you saw the other person do during this project that impressed you? Because we all, we are in a room every day together. I called it bootcamp for a while. So Al I'm going to go to you and ask for you to say one thing that's impressed you about what Ben has brought to this project.

Stevens: Well, I know you prepped me with this question before, and I had trouble because there are so many things that Ben has done. Without Ben's glue that worked across the clinical and technical teams, we wouldn't be where we are today. And, that has been critical. Ben's reaching out to not only the internal clinical members of the team, but our external users and spending hours with innovation partners going through in detail what we were thinking about and bringing that feedback back has been equally critical to the overall team as the relationship between the clinicians and the developers, that we really had a third partner, which was our innovation partner early users, but that Ben was instrumental in bringing back all of that information, communicating both ways.

Eisenberg: Fantastic, completely agree. I definitely think the glue is a good, good nickname, for Ben. I know I have to try not to lean on him too much just because he can bring so many skills. Ben, I would ask the same question to you about Al. 

Hollis: And thank you Al, I appreciate it, I appreciate that. I would say, you know, tenacity probably is the one word I would, I would use. It’s not always easy having a team of such disparate, strong willed folks. But Al, you know, Al has been really able to drive that team forward, hold that team together. And also, you know, just inspire a culture of persistence and experimentation. And that is not easy to have that focus on allowing failure, allowing, you know, going down the wrong path, pivoting, and pivoting quickly. So that's what I would call out because I think it's one of the few teams I've had the pleasure of being a part of that really embraces that spirit. And it's not easy to do even in a startup, but much harder to do in a large corporate organization. So I think that's probably the thing that's impressed me the most, is carving out the space for innovation within a large, you know, sort of, you know, bureaucracy essentially, as as all corporations are. That's a great reflection.

Stevens: Thank you.

Eisenberg: So on that note, I'm going to give you each one final question before we wrap. So, I'm looking for one word from each of you to describe this journey so far. Ben, I'll go back to you, and then we'll end with Al.

Hollis: I would say inspiring, mainly because of the way this team has been able to adapt to new technology, to innovate, but also to really, I think, enjoy being together and working on a challenging problem that has real positive benefits in the world. And, you know, you don't always get to work on things that are inspiring in your, as your job. And so it's been a real pleasure to be able to do that with such a great group of people.

Eisenberg: Fantastic. Al, how about you? 

Stevens: Well, this is a hard question. The one word that comes to mind is adventure. And the reason adventure comes to mind is, you know, first, if someone said that I'm going to run this project as an adventure, if I was trying, sponsoring the project, I would say, you know, no way. I want a project, not an adventure - That is not, that does not work! But if you think about what an adventure is, you're trying to learn something, you're trying to move forward. You're willing to take risks. You're willing to fail in certain parts. You typically do have a goal at the end. And I think this project has been characterized by those aspects that we didn't know where we were going to end up. You know, we entered this world and we have learned a lot. That's what happens during adventures. And we have made a lot of progress, and have developed experiences and knowledge that's going to help us move forward.

Eisenberg: Fantastic. That's a great one. 

Hollis: Yeah. 

Stevens: Yeah, absolutely. 

Eisenberg: Well, thank you both for taking the time to join. I know we are still building and moving quickly, so appreciate the time. Appreciate your reflections and I'll let you get on with your day now.