[Sydney] Good day to everyone joining us and welcome to today's X talks webinar. Today's talk is entitled the RWE ROI Series: The Transformational Value of Real World Data for Drug Development and Regulatory Decision-Making. My name is Sydney Perelmutter and I'll be your Xtalks host for today.

Today's webinar will run for approximately 60 minutes. This presentation includes a Q&A session with our speakers. This webinar is designed to be interactive and webinars work best when you're involved, so please feel free to submit questions and comments for our speakers throughout the presentation using the questions chat box and we'll try to attend to your questions during the Q&A session. This chat box is located in the control panel on the right hand side of your screen. If you require any assistance please contact me at any time by sending a message using this chat panel. At this time all participants are in listen-only mode.

Please note that this event will be recorded and made available for streaming on xtalks.com.

At this point I'd like to hand the mic over to Evelyn Pyper who will introduce our speakers for today's event. So Evelyn, you may begin when ready.

[Evelyn] Great! Thank you so much Sydney and good morning to everyone in North America attending and good afternoon to anyone joining from Europe or the EMEA region. I'm Evelyn Pyper and I lead Evidence Strategy at PicnicHealth and I am really thrilled to launch our picnic Health 2023 Webinar Series, The RWE ROI.

This overarching theme really came about from the fact that even though real world evidence has become well-recognized as part an important part of the drug lifecycle, many of the conversations that we've overheard happening about RWE, whether at conferences or webinars, still felt very surface-level and didn't necessarily convey the critical return on investment or risk of an action regarding real world evidence. Throughout the rest of this webinar series, each session will convene a unique panel of global stakeholders to explore the RWE ROI from a variety of perspectives.

[Evelyn] So today to launch things off we have a very distinguished panel of speakers to launch this webinar series. Our focus today is on the emerging and really exciting space of real world data for drug development and regulatory decision making.

To introduce our speakers I'll start with Jesper Kjaer. Jesper is Director of The Data Analytics Centre at the Danish Medicines Agency, and Co-Chair for HMA / EMA Big Data Steering Group. Jesper has 20+ years of experience in data management, analyses and data visualization, having previously worked in academia and the pharmaceutical industry.

He has headed up activities in EU Framework Programmes, TransCelerate Biopharma. Jesper is Co-PI of PHAIR: Pharmacovigilance by AI Real-time analyses, applying FHIR resources to healthcare data in DK for real-time safety surveillance. He is involved in EHDS pilot work and is an EMA DARWIN EU advisory board member.

Welcome Jesper we're really really pleased to have you. Thank you so much.

We also have with us today Noga Leviner. Noga is the Co-founder and CEO of PicnicHealth, a digital health company founded on the belief that empowering patients to gain control of their medical data is the critical first step to improving lives and health outcomes and driving scientific advances. PicnicHealth has pioneered the use of advanced human-in-the-loop machine learning algorithms to abstract key endpoints from medical records at scale with improved accuracy and efficiency. Noga is a vocal advocate for patients’ rights to control their own health data, having founded PicnicHealth in 2014 to help patients manage their medical records after personally facing the challenges of being a Crohn’s disease patient in the US medical system. She has spoken widely on the subject including at the White House. Welcome Noga.

[Noga] Good morning.

[Evelyn] Thanks so much. We will also have joining us today shortly Najat Khan, but I will save her bio for  when she's on our call and will keep you on the edge of your seat for that one. So to kick things off, we want to you know, before we get into a deeper Q&A to really understand the perspective that each of our speakers are coming from, because we've intentionally convened folks that are coming from slightly different angles of this topic. So, a common thread that connects all of our panelists is really a dedication to ensuring that real world data is living up to its potential.

Historically we know that real world evidence has largely focused on more downstream post-authorization used but in line with today's theme each of you are contributing to important upstream applications of real-world data for pharmaceutical Innovation. To start off, I'll just ask each of you to share briefly,  a less than five minute overview of the perspective you're bringing to this topic. And second, what your priorities have been over the past year. I will start with Jesper to provide your regulatory perspective

[Jesper] Well thank you. So I think it's important for me to start with saying that the randomized controlled trials or clinical trials are the gold standard right. We're not from a regulatory perspective stepping away from that. But we're realizing that real world data and evidence is really a very important supplement, and sometimes even we can observe it is probably the only real possibility. You often find yourself in situations where small populations, unethical situations that are not really suitable for randomization will lead you to using real world data to generate your evidence. But it's very important for us to look at this not as a binary situation, but something where each of them brings their own value.

Now there's a long history of actually using real world evidence. We may have called it something else in the past, like observational studies. Some of them more prospectively than secondary prospective data collection, more than secondary use of existing data. We have a reality around us that is changing where more and more health care data is born in an electronic way and  that gives new opportunities and it would be an absolute waste not to use those opportunities to get better insights into the effects and side effects of treatments and also the value of medical devices. There's in particular the use around side effects obviously, for very good reasons. You will not be able to uncover every single possible side effect in the clinical trial. We can do as much as we possibly can, within the limits and reasonable design of those clinical trials to uncover that-but there will be situations where actually just running the clinical trials will put us in a situation where we're looking with a certain focus lens into the population that will ultimately be treated by the new medication. And for that purpose, over the years we've had the instruments of post approval effectiveness and safety studies that, in many cases, have been the real world data so to speak that have been the foundation for us to make further decisions and evaluations.

Now with these new opportunities, we are strengthening those capabilities by using that data for that. And we're starting to see examples where real-time surveillance after approval actually can be a real thing in the data pools that are available. But it's very important for these data sources that we're able to declare them, that we're able to describe them. I would assume that most data sources can actually be used, but they may not all be well-suited for the scientific questions you're going to look at with them. And-not going into the details of what distinguishes one data source from another-the whole understanding about that not every data source is equally useful, it's just very important to have at hand. And even with the best possible data sources that we know of that have been extensively used and demonstrated their validity and robustness over the years, it’s still very important we actually have the meter data in place to understand these data sources.

We basically need to understand what happened with the data as it was collected, stored, and handled so we can truly understand that what we're looking at is not an effect of circumstances with data handling or new policies but the real situation with the treatments we're looking at. And the real situation we would like to understand is in a real world setting: what are the situations-in all sorts of populations that are now getting the treatment-whether that's by choice of the treating physician, whether that's new indications, whether that is in some cases what we see that research is leading to an off-label use. We would like to understand that continuously, but more importantly, in the real situation of comorbidity and polypharmacy. Really understanding what we couldn't have-all possible outcomes in the clinical trials that we need the real world data to understand.

We'll also need the real world data up front to truly understand disease progression, standard of care, and all of that really is underneath of developing new medicines that can inform us in our decision process. Then I think what we are aiming at with some of the initiatives in Europe with the Big Data Steering group of building both a DARWIN EU Data Quality Framework, the training we're doing and also the data standardization strategy, is really to think about data quality and a standardized approach to this. One thing I think we still need to develop and think about is how can we actually not only make passive secondary use of the data, but really look at how do we move ahead to a Learning Healthcare System where prospective data collection is something that is going to be even more helpful so we ultimately can avoid some of the challenges with secondary use. In clinical trials we have the protocols to really define what to do and then describe whenever we deviate, and why do we deviate, so we can truly understand what is cause and effect in the data we're looking at, but we are not left without opportunity with the secondary use of the data.

That's why we are proposing the Data Quality Framework to better describe that. To have the metadata to understand the ability to use the data and the ability to have missing and wrong data and understand why did that originate. [This is] something we can actually deal with in a clinical trial and prospective protocol-related data collection as well. So that's something I hope will continue down that path, and I'm confident with the initiatives we have right now in Europe, we're going to see a lot of real world evidence being used in the decision processes and primarily fueled by the fact that as you establish the DARWIN EU system, you build the quality framework, and then you expand over the years with European health data space the infrastructure is there. The notion that this data will also be used for developing medicines, both in the pre-approval and post-approval phases, will lead to a different way of collecting that data and using that data for that purpose as well, which is something I'm very much looking forward to happening.

[Evelyn] Thank you so much. I think hopefully I can speak for everyone when I say that the model that you have developed is something I think many other regions would be smart to adopt. And certainly as I sit here in Canada, with our, you know, fragmented data ecosystem, it's encouraging when I hear of others trying to follow in the footsteps of the EMEA when it comes to this sort of data infrastructure. Before I pass things over to Noga to introduce her solutions provider perspective, I do want to introduce Najat who has joined us.

[Evelyn] Welcome to Najat.

[Najat] Thank you for having me.

[Evelyn] Thank you so much. We're really pleased to have you. So for everyone, Najat is the Chief Data Science Officer and Global Head of Strategy & Operations for R&D at the Janssen Pharmaceutical Companies of Johnson & Johnson. In this combination of leadership roles — unique in the healthcare industry — Najat is responsible for overseeing the Janssen R&D strategy, pipeline and portfolio optimization and investment, and for fully integrating data science and digital health end-to-end across the pharmaceutical pipeline to drive transformational innovation for patients.

Welcome to Najat. We're looking forward to hearing a bit more about the perspective you bring to this particular topic in a moment.

[Najat] That's great.

[Evelyn] Great. So Noga, I will pass things over to you, to tell us a little bit about the perspective you bring in terms of what PicnicHealth is and what your priorities have been over the past year.

[Noga] Sure, thanks Evelyn. I'll start with a quick introduction to the company. PicnicHealth is a digital health company that builds deep real world datasets. We are able to get this deeper data in two ways. The first, which I'm sure I'll end up talking about quite a bit, is just working directly with consenting patients. We take a very patient-centric approach, meaning we have a direct relationship with the patient, so we can really go get all the data that surrounds that individual, including from all of their medical providers, but also incorporating, you know, both historical and longitudinally going forward, but also incorporating their reported outcomes and points of view. So that's one key part.

And then the other side is actually from a technology perspective. Having the human-in-the-loop machine learning model that you alluded to earlier in the introduction, which combines the state-of-the-art machine learning with human curation. So while the patient relationship gives us that deep complete view, our machine learning combined with human curation allows us to then go really deep within the records and extract those key clinical endpoints, that really key information, that's in unstructured text in a way that's both high quality and meeting the quality standards-that I'm sure we'll continue to hear a lot about-but also that's possible to scale. That's scalable thanks to the help of the machine learning, so what you end up with is: a very complete, clinically-rich view of the patient, that's longitudinal both retrospective and then of course prospectively following the patient and being able to get at their patient reported outcomes. And because the data doesn't come from a particular site or  type of facility (medical facility) we actually get really quite good representativeness because we are able to include diverse patients who have a variety of comorbidities, disease subtypes, and treatment histories. Because the process to consent and to participate is meeting the patients where they are, like in their own communities at home, it's a much lower bar to participate and sort of lower requirement than what you would see in a clinical trial.

So I'll comment and say that this data, in light of the comments we just heard Jesper make, I'll say this data doesn't replace broad data for something like surveillance. It doesn't replace that really broad, kind of shallower data for something like surveillance, but we see a lot of demand for this richer data, richer real-world data, as well coming from Pharma functions across the product lifecycle from R&D to Market Access, Med Affairs, Commercial teams. And of course relevant today through our work with our life sciences partners, we are increasingly seeing real world data leveraged to understand disease earlier in the value chain and optimize clinical trials.

And you asked about priorities so I'll touch on that quickly. In the past year we raised our $60 million dollar Series C funding which is fantastic. Going into the current environment, we are growing our research program portfolio, connecting with new patient populations, and always growing and continuing to build our industry partnerships.

[Evelyn]  Fantastic! Thanks so much, Noga.

[Evelyn] And finally from the pharmaceutical perspective we have Najat. We heard from your introduction that there's quite a bit that falls under the umbrella of data science and R&D strategy. I'd love to hear about where your priorities have been in terms of real world data specifically and where you're coming from for this topic.

[Najat] Sure, that's great, Evelyn. Thank you so much again and to PicnicHealth for having me here. As you can see I'm trying to get better lighting, but hopefully you can hear me fine, which is what matters the most.

First of all, I want to say, both panelists, the comments are really spot on. You know, maybe just taking a step back from a pharmaceutical perspective and the perspective really focused on how do we make better medicines for patients, right? I mean that's, at the end of the day, the insights: things we can do, like leveraging verbal data, that we couldn't do before, or things we can do better. All are driven to that common purpose of translating it into insights and then a medicine that's for patients. I would say the way I think about real world data is not even just in clinical development. Starting from the very crux of the question that we're trying to answer, what's driving the disease, right?

I mean I think that's where it's really challenging. Like you know even today how much we understand about disease biology. If you think about so many diseases, Alzheimer's, Parkinson's, they're not named after the driver or the causal effect of whatever is driving the disease, but much more so based on the physician that identified the syndrome. So that in itself tells you that there needs to be a deeper understanding of disease biology. So the way I define real world data is of course claims and EHR, but you also have omics, like transcriptomics, genomics, and images, everything that's not clinical trial data. So if you think of the totality of understanding a person's journey, what drives disease, you need to look at the totality of data. And so we are actually leveraging that from the very first part, which is better understanding the drivers of disease and redefining disease, as a result of it.

So as an example, a lot of the audience probably knows about these longitudinal data sets-for instance, Noga was mentioning too-called UK Biobank or Our Future Health, I mean there's so many. But essentially it's longitudinal. You have all of these different data points collected well, good data quality, and what that has allowed us to do is to be able to stratify people that have severe depression to patients that have severe depression based on their sort of genotypic and phenotypic architecture. So really driving precision medicine more into an area like depression, which let's face it, the medicines that we have today work but we could do better. I think we can all agree that we could do better and that insight is then translating into how we design our programs for depression. I wanted to use depression as an example because we talk a lot about precision medicine and oncology but we need to bring that in Immunology and Neuro and so that's just one example.

Another way that we're using real world data a lot is, let's say you figure out what's the target that's causing that protein and you figure out the molecule you need to actually modulate it. The next thing that comes up is how do you run the right trial and design it the right way. And Noga spoke a little bit about this as well, which is, instead of relying on publications or instead of relying on multi-center sort of analyses of the patient's journey, you could now use real world data across the board. Across all of the patients that actually have a disease, then go back and say, "What were the risk factors that were driving them?", and that actually helps you design the protocol in a much more effective way. So you can do certain things such as inclusion/exclusion criteria, you can simulate it using real world data to say, "Okay, is this actually reflective of the patient journey in the real world?", and "Why is that important?".

It’s actually because it's important when you start recruiting the patients. You can't operationalize something if it's not pragmatic in terms of what's truly happening in the real world. And we've done that, like for one of our vaccine programs. For instance, we used 100 million patient lives-de-identified of course-working with external partners and then really understanding what are the risk factors that they'll get this certain infection. And that actually was then translated into the protocol design itself, working with our clinicians and now that study is running in Phase 3. It helps helped us enrich the trial a lot more with the right patient that would be impacted, which is the patients, but it also helped us ensure that a trial is much more efficient so we can get to the data that we need and then hopefully the therapy that we need, knock on wood, for patients as quick as quickly as possible.

And maybe the last example I'll mention is also just a lot of the viewers here probably saw some of the  guidance on external control arms that came out yesterday from the FDA. Super helpful, because the way we think about real world studies broadly-and ECA is a part of it (external control arms)-is you can leverage that to really contextualize what's the right standard of care, how are patients doing in the real world with the physician's choice versus the active arm of a trial.

We're not just thinking about it, we've actually leveraged this for our regulatory submissions, working very closely with the FDA. Early pre-specification conversations are super important, so you're getting feedback on your protocol, on the data elements being super transparent about the pros and cons, all of that. The thing I want to emphasize is no data is perfect and no guidance is perfect. There's always a case-by-case discussion, but this is where early discussions and collaborations and transparency and rigor in what's done becomes supremely important and that's something I want to underscore. Like in my organization, we have the highest highest bar because there's a lot of real world studies, but to do it well takes a totally different level of talent and we'll talk more about that and capabilities later.

And then last thing, but not the least, is also the space around what I would say is clinical trial recruitment. We're using real world data to understand where the eligible patients are and then go to the patients, open our sites where the patients are verses the other way around. And we've seen tremendous impact in terms of both how effectively we can recruit, but then also ensuring that our trials are representative of the patients, that we will treat from a demographic perspective, from a diversity perspective, etc. Using decentralized trials and so forth.

So as I have this monologue for the last couple of minutes-and thank you everyone for bearing with me-the point to underscore every single aspect of how we figure out what was causing the disease, to designing that all-important trial, to executing that all-important trial, to generating that all-important evidence would becomes critical for access that ultimately helps to get our medicines to patients. Every single aspect of it is being disrupted positively using real world data.

[Evelyn] Thanks so much. Yeah, all really salient examples. And I know you mentioned that it felt like a real treat yesterday, that the day right before our webinar, the FDA released the guidance on external control arms. I think it's just reflective of  month-by-month, things are developed. Like over the past year, the amount of guidance that's come out, the amount of real action around providing industry guidance, but also guidance to companies like ours, like PicnicHealth, where it's really showing this is not just a nice to have anymore, it really is something that the expectation is high quality real world data as part of an evidence submission.

[Najat] And Evelyn if I can just maybe underscore one thing that you mentioned. You know all the examples I mentioned, it's not something that we just do internally, right. Yes, we build the algorithms, the methodologies internally, but the datasets and the partnerships, we have accelerated a lot because of the work that, you know, companies, young companies like PicnicHealth, and others are doing. It's a lot of hard work. Let's just put it this way. To take a lot of the data that's fragmented, not always very clean, not standardized as guidance is coming and pulling that together, creating the cohorts-Noga was saying curating a lot of the work in the structured and unstructured datasets-that all becomes extremely important. So, it's a partnership across regulators,  startup companies, and also pharma and biotech companies. It all has to come together. We couldn't do it without each other. It's exciting to hear the examples, but the foundation of it is the work that's being done across the board. So I just want to acknowledge that.

[Evelyn] Thanks so much. Really appreciate that call out. And maybe we'll shift gears from our big picture, all the things that we're all tackling, into a bit more of a slightly personal fun question. So a career in this space of data strategy, data science, applications of advanced analytics really requires a unique set of strengths and capabilities that each of our panelists possesses. Arguably these skills are valued and transferable across virtually every sector today. So question for each of you: Why healthcare? Why you do you find yourself here in healthcare using your skills for this type of mission versus you know financial sector or other technology sectors?

I'll start things off with Najat. We'll bounce back to you.

[Najat] Sure. Why not healthcare? I mean, listen, I'm a bit biased maybe. A quick backstory, not to bore anyone, but you know I grew up in Bangladesh and then I grew up in the UK. I came to the US when I was 17. So really lots of continents in a short period of time. And the thing that always moved me from early on, and it helped that my parents were both physicians, is the fact that when you see somebody get better, person, animal (I'm a big animal lover), that feeling is unmatched.

Like I even get goosebumps as I think about when somebody actually, it's not just the patient, you see the impact on the family. Early on, I recognized that was something that would just get me out of bed, like that's my passion. The other thing I also recognized was that the pain and suffering when you actually have a disease, whether it's for the family or the patient, that's common, across any country, whether it's a developing country like Bangladesh, or you're in the UK, in the harshest area, or in the US and some of the communities that are underprivileged.

And the equity in terms of health care is something that we're not where we need to be. So I think I knew early on just based on that. It's just something that's unwritten. I also growing up did a lot of work around non-profit work going to the fields that I remember I was excited about it. My dad was like if you're excited about something just go to the entry level and just experience it right and it was tough and I didn't know how to speak the language and there's so much, but the barrier and the kindness and the empathy that comes from it. I just think it's not something, anything I could feel in any different sector.  And then COVID just puts it on another level right. Like the reality of it, as we saw, that was the most important thing. Health became the most important thing and so just fast forwarding.  

Then the next question was how do you impact and you mentioned data is so important but I mean-I don't know Jesper, Noga maybe can comment on it too-like 20 years ago this wasn't even a very well established field. I remember when I was doing my undergrad as an international student which means you have to keep your GPA super high or else it's a problem because I was on a scholarship and stuff anyways. I did  all of the sciences like physics, chemistry, bio. I also did computer science and I also did econ. So it was just this weird plethora of different disciplines and I remember so many people asked me “Najat, why? Do you just like pain? Why are you doing all this stuff?” And I was like well, you know, I could see the computational aspect would become more important as more data is generated and just like things that we cannot compute in our minds right and science was at the core of what I cared about: science and medicine of course.

The reason I went to a liberal arts college and did econ, I recognized when I was growing up that unless you take all those great ideas and make something out of it like a business that's sustainable you can't really have an impact in a sustained way for patients. So that was the confluence of it. It came like a very organic way and when I talk about it retrospectively it sounds so cool, but honestly at that point I was gonna say, "I'm interested. I think this is going to be interesting". So, then when I did my PhD at Penn I was very lucky. I had a crossroads. I could pick an advisor that was a very core organic chemistry, like you know publish, publish, publish, and then I had another advisor who was new not, tenured, so higher risk but was much more into interdisciplinary approaches. Long story short, my PhD was a confluence of working on physical modeling on what molecules might work and there are so many colleagues in the computer science department that helped me. I learned from them. Then actually coming to lab and making it with my own hands as an organic chemist. So I was developing molecules for detecting early cancer. Early detection of cancer and also therapeutics and then actually working with folks in the Penn med school to translate that into in vitro and in vivo work and that was a very sort of non-traditional PhD but it was very application-focused and had a PI that actually was like, "okay if you do good work, if you don't we're gonna have a conversation." So the reason I'm saying all this for folks that are watching is it's not something I decided on but I think as you progress, now thinking about data science and AI, machine learning, RWE is great, but the question is what do you apply it to. So understanding multiple disciplines of what you applied to that is going to actually have an impact and not just doing it because it's a cool algorithm or a cool methodology is super important if your core mission is to have impact on patients. So I'll just say I could do it in finance but I don't think I'd be this excited if I did it.

[Evelyn] Well thanks so much. I’ll pass things over to Noga.  Why healthcare for you? I know Najat just mentioned creating sustainable companies and that felt like a really good segue to pop it over to you.

[Noga] I think in some ways I think my experience parallels yours, Najat, but actually it's kind of the opposite because I got into this a little bit later in life. I actually also will just share my personal story. I got into Healthcare and started and founded PicnicHealth because I'm a patient myself and until I had the experience of being a patient, I wouldn't have thought of applying the skills that I had built over my career to healthcare. I think it's like so trite, like so obvious that it goes without saying, people always say like if you don't have your health you don't have anything, but when you're in the situation where you're actually dealing with that there's just like no substitute. There’s nothing more obvious than if you don't have your  physical, your mental well-being, you don't really have anything else. I was actually a patient and I was diagnosed with Crohn's disease in my 20s and it was just looking around me at the landscape. It was just so obvious that if we could act like from the experience of being a patient and how divorced I was from the research space, how hard it was for me and frustrating it was for me to get a hold of my own data that if we could just bring patients, that there was a sustainable business to be built that would benefit patient and so clearly benefit other parts of the healthcare sector.

I think the bar for improving things in terms of the healthcare experience for patients and ability to participate is sadly so low. There's so little that's happened. The world is changing I think things are getting a bit better, but I think at the end of the day being here in Silicon Valley, you feel like there's people who are like spending their whole careers optimizing l some little thing, like a little button and then you kind of look at the experience patients are having in healthcare and the way we're using data in our system and it's like these broad swaths of opportunity. For anyone who's had any kind of personal experience you can just see that there's like massive massive opportunity for really big shifts in impacts that you can have. That's really kind of palpable.

[Evelyn] Thanks so much, Noga. Jesper, how about you? How did you find your way into healthcare?

[Jesper] By failure actually. I failed studying veterinary medicine  back in the 90s and then I realized while I couldn't really learn another anatomy textbook by heart, it was during the whole internet development and taking the IT route was just interesting. Just by chance, one of my colleagues said I have this database at one of the hospitals that needs a little bit of help, "Can you go and meet up with them?" And then ended up in a research group for the next 13 years. It turned out what we did there was really impactful. There was observational data on a global scale. The DID study was actually funded by industry through the FDA to look at adverse drug reactions for HIV treatment. Obviously that suddenly became really really meaningful as you had that opportunity it also gave you an opportunity to suddenly realize how closely connected the world is in this.

I remember some of the work we did. We actually went to the NIH in Washington and presented to Anthony Fauci about data models and what to do on a global scale for a network that still exists today. And then getting that interest of industry, and now the regulatory world. I just got soaked into it and couldn't let go of it. To me observational data, the real world data, has been part of my work for the past more than 20 years now, so not really getting out of it. It just seems to be coming back in new wrapping. And more of it by the way. So a lot more data than in the past.

[Evelyn] Well I think we're all glad that you're here, that the three of you are in healthcare…And probably we won't let you leave because there's more work to be done! Jumping back to you Jasper,I saw you speak about DARWIN EU at a conference last fall. It was fantastic. Since then, I saw that the EMA announced the first eight data partners selected as part of the Phase I of DARWIN. These seem to include both public and private institutions from across Europe including Spain, Netherlands, Estonia,Finland, France and the UK. From your perspective, if you could distill down, in the time that we have, what are the most important considerations for assessing or selecting a data source or partner whether it be for the purpose of DARWIN or more broadly?

[Jesper] I mean DARWIN in particular actually has a connection office that proposes this to EMA and The Advisory Board as well. There are lots of factors to be considered, maturity of the data source, the quality management system of that, the whole procedures, and ultimately, at the end of the day, also that you can demonstrate scientific value from that data.

Then this is a network of data sources, your capability to connect with the common data model and then interact in a collaboration around this is obviously also important. I think there's a number of factors, but if you would take one down, is really the scientific question driving the selection of the data sources because we need the relevant data sources to the scientific questions we actually see in the regulatory work. So, you'll see a bit of dynamic around this as we expand into primary or secondary particular disease registries as well. We're talking really rare disease. We have some of the reference networks in Europe we can utilize in a different fashion but it's really true to make sure we get the appropriate data sources to the scientific questions and then obviously data quality and Quality Management Systems around that matter.

But ultimately we are fortunate to have many data sources where we can pick from. So, I think the choice is really to give everyone room and then see it develop over the years to come.

[Evelyn] Great, thanks so much!  Najat, given you have global oversight, certainly at any given time I imagine you're thinking a lot about what's happening in Europe, what's happening in the U.S and elsewhere. In terms of data systems, historically we're often very confined to these geographic barriers and I'm curious if you're seeing promising progress in how real world data can be used or is being used today across diverse geographic contexts? Any examples of that?

[Najat] I mean, look there's definitely progress being made with different approaches in different places. Maybe I'll break it up into two ways. One is true regulatory grade real-world data and then there is real-world data non-regularly grade. I think for the number two bucket there's quite a bit of progress being made, especially in regions like Brazil, in the LATAM region, in China, and others as well. Sometimes people ask me, "Is that even important?? It actually is. It depends on what question you're trying to answer. If you're trying to answer the questions around-Noga mentioned this, Jesper mentioned-understanding the patient journey. Understanding it early on. What's the standard of care there? So it helps you for the design. I was saying the protocol in a much more thoughtful way with a global mindset, which is definitely critical for us. Also if you're trying to recruit patients, just understanding which sites patients are going to. From a site collection perspective there's a lot of opportunity. I think Noga said, in terms of the bar there's a lot to be done. I always say the flip of it is only 10% of what we make has success. So, there's huge room for improvement. I think from that perspective, especially with LATAM, also some of the Gulf Countries, we are starting to see that they're actually trying to aggregate a lot of their data sets, connect it, link it and so forth. Maybe not at a national level, but much more at least regional levels or key hospitals. So, it's a start.

Let me think about more globalization and having more diversity in our trial, that's definitely helping, so that's one example. I would say from a regulatory grade perspective it's still really…. like I was mentioning before, a lot of the work in the U.S. I think a lot of the funding that's really gone on from VCs and private equity funds, etc. Congratulations Noga on the recent raise. It's really helping us, in the last few years, push to having higher regulatory grade rich data. Because at the end of the day a lot of it comes down to the exact question you're trying to answer.

I think in Europe, Jesper can say more than I can, but with DARWIN and a lot of the other networks it's definitely getting there as well.  Ex-U.S, some of the other regions, I don't think they're at that regulatory grade yet. If I can be just really frank, there are some changes, like in Singapore, there's some really good data sets from Taiwan, but large countries because I think the step one is just getting everything together. Then next step two is that next layer of maturity of actually making sure that your data quality, privacy considerations…. Alot of the guidance is also still evolving in some of those regions as well. So try to be practical about the answer but it's definitely moving in the right direction. But what I want to emphasize is even what's happened today, it can be really really useful in terms of how you develop it.

[Evelyn] Definitely. Shifting gears to Noga.  I think you might have a unique perspective on this coming from PicnicHealth. Are you seeing anything today as someone who's working with life sciences partners that suggests that these attitudes around data crossing borders are changing or should change?

[Noga] I'll leave the whether they should change aside for the moment, but I will say that I think we're starting to see some encouraging signals of progress that there is for openness to using real world data across regions. At the core I think people recognize that, more and more, every data point counts and you just have to find patients where they are.

Among our broader portfolio of therapeutic areas we work in we really see this the most in rare and ultra rare diseases, where every data point, every patient counts. It’s particularly important and what we've found is that what's most important is the ability to be flexible to adjust to what matters in a particular region or context, to be able to get that really complete picture. Then we found that being able to, with low activation energy, go back in and change the data model to meet what's important in a particular region or context, customize it to align with what they're looking for, it really does a lot to give comfort rather than just having a static data set that has to be used in the way it was designed for another region.

Ultimately I think sponsors are using our data to support evidence packages and regulatory submissions both in the US and increasingly across borders as well.

[Evelyn] Thanks, and from this PicnicHealth perspective, as Noga knows, we're often working with real-world data and evidence champions from their respective organizations as opposed to you know the naysayers or maybe traditional trialists. The challenge I think in PicnicHealth is less about convincing folks the value of real world data and these various novel applications, but more aboutarming our main champions with the information. That'll help them have broader internal buy-in.

[Evelyn] So a question, we'll start with Jesper and then move on to Najat.  You both champion initiatives that can be considered innovative in their use of data and data sciences. What's your experience been with navigating and securing internal buy-in for real-world data projects and fighting that good fight for real-world data?

[Jesper] Often when it's maybe outside the status quo,  I think what has really been the driver with us has been a strategic approach from the top and then downwards. There's been something that we decided to put into the European strategies, in the Danish medicines obviously also for the past 10 years in our strategy and then I think it obviously helps if you're in an environment like the Danish, where doing registry-based research to inform how you do healthcare and treatment and so forth is just baked into the way you do things.

That has been a major factor in making that change and then through demonstration projects, step-by-step to actually do this in our procedures and our processes. To bake in the use of data in a different way. That's why we established the Data Analytics Center, to have a core function of expertise that would collaborate with the other skills in our agency to use real world data in decision making processes. Whether that's in our pricing department, whether that's in our safety surveillance, whether that's in our approvals, even to the point of our HR system, we can actually do stuff with data analytics matters. Just bake that in everywhere, that starts to change the paradigm. That's the same mechanism with the other agencies in Europe and it's the same mechanism within EMA. It's really something that we from a strategic point of view, have decided to do and that really makes the change.

[Evelyn] That's great. Najat, you've probably seen changes over time but today, are you still needing to convince the value of real world evidence? Are we past that stage? Can you push the boundaries a bit more around what we're doing then with that real world data and evidence?

[Najat] What I would say is, right now, where we are there are a couple of things. I think it's very similar to what Jesper said. It started very much with, will real world data actually replace RCT? That was the big thing if you think like four or five years ago. I think that was almost as over ambitious an aspiration which was not based in reality. The reality is clinical trial data is extremely important, but we can always say there's more data that you need to have more holistic evidence generation. It’s an “and,” it’s not an "or".

So step one was just to shift that at least, in my experience and like Jasper said, it started from the top. Our head of R&D just had great foresight in terms of the impact of data science that can have end-to-end. Then our CEO, Joaquin Duato, I mean he has said the combination of science and data science technology is going to be what's really propelling us in the future. So, I think having that foresight investment support from the top. Then the other thing we also did which was really important is building a team because to know what good looks like and to actually do great work and to play in a professional arena, you have to have professionals at the end of the day. So, we built it as a function, as a new business unit that actually reports through me to the EC eventually.

The thing that was important as I built out the team, was to get that grassroots because you don't want just a top-down for a large organization. You want the grassroots support as well. You need to have demonstration projects, back to what Jasper was saying, but focus in areas that are actually going to make a difference and in a time frame that's not five years. That's why we started in Clinical Dev and not Discovery because they're taking so long, just being super practical. Everybody can get behind that question, like all different groups, but the other part was actually having bilingual talent that understands clinical science. A lot of them have backgrounds in oncology, neuroscience, and so forth, but also folks that are appointed experts in real world data, AI, machine learning, and so forth.  That creates a common bridge in languages so that you can actually have a discussion at the grassroots level. How would you solve a lot of these problems? For where we are right now, tremendous progress. We started building this team two and a half years ago. It feels like much longer than that. We have data science fit-for-purpose use cases across every single clinical program, so it's totally done at scale today. You'll get the odd questions, "Do we need real world data?", but we have enough demonstration projects where it started off as, "hey let's keep it  for an optionality perspective," but then it becomes really important as you go towards regulatory submissions.

The other thing is it's been done with great care and rigor, not just internally but with our external partners and also with great SMEs. I also always have the team actually review it with third-party SME, so it's like you never want to be drinking too much of your own Kool-Aid. I always say you always want people to look at it. Like maybe this is a grad student in me and I’ll always poke holes into everything because that makes the ultimate output better.

There's a lot of support, but there's always priorities in an organization so it's continuous. It's not ever done. There's always priorities and if there's one bad real-world study that comes up, everybody regresses and that's what we have to be really careful about. If it's not done the right way, same thing with AI algorithms and so forth. That's why I always say we have to work together as a team, internal and external.

[Evelyn] Well thanks so much. I imagine, too, as the suite of capabilities that real world data can be applied to is ever-growing, I'm imagining that there's always going to be that iterative conversation around, "Five years ago we weren't using real world data for X, but now we are." So,  how do we make sure it's the quality and the output is what we need to see?

On that note, because we're coming down for time, and we do want to save time for questions from the audience at the end, I'll ask for maybe a rapid fire round of responses from our panelists on this last one. With all these advancements in data science and digital health, it's transforming how we're using real world data as all of you have spoken about today in clinical research and in clinical care. If you had to pick one data or digital development that has you really really excited for the year ahead because of its potential to transform our understanding of a disease what would that be? Noga, I will start with you.

[Noga] That's a great question. I will say having now worked in the sector for 10 years, I think almost every single year actually when I got into this…. I was re-watching a video the other day that I had put together when we were just starting 10 years ago and my co-founder and I were saying, "there's been a regulatory change and within a matter of a year, patients are all going to have access to all their data through APIs." This complete transformation is happening and that was basically the last time that I was optimistic about that. I learned my lesson very quickly, but there have been these glimmers of hope along the way that were actually happening, that there was progress. That regulatory change is going to make a difference and I think the reality is, we have learned not to be overly optimistic. And yet I actually think what's happening right now and the changes that 21st Century Cures Act is bringing on is incredibly powerful and exciting. It's really like we're going to be shifting the balance of power to ensure not only that patients can get their data but also that they're really empowered to choose how to share it.

Ultimately, what that means is fewer barriers to access for patients and much more real-time data, which I think not only matters for patients and their own direct experience navigating the healthcare system, but also for the kind of life-saving clinical research that they want to participate in and choose to contribute to.

[Evelyn] Great and I think a good segue to Jesper, in terms of a regulatory example from Noga. What has you excited as a regulator yourself about this year ahead?

[Jesper] Oh no doubt,  the scaling of DARWIN EI into our regulatory processes is the biggest. But what I'm also seeing is that in our own little country, we are taking learnings from Covid-19 and the use of real-time surveillance to the next level together with computer scientists and building artificial intelligence machine learning techniques on top of that because ultimately the next pandemic may unfortunately just be around the corner. So, if there's an opportunity to learn from that. I'll see some of that effect this year already and we're looking forward to having a kind of a new tool available in our toolbox.

[Evelyn] Great! thanks! And Najat, how about you? I'm sure there's lots that has you excited for the year ahead, but what would be the standout for you?

[Najat] What I'm really excited about is the fact that thanks to ChatGPT, which still needs a lot of work, AI is actually becoming much more mainstream and people are starting to get that this is here to stay, which I think is actually, if done the right way, is really good for everything we're talking about. Because I think one challenge that comes up is sometimes-I'm just thinking for the broader audience-it's hard to understand how all of this is going to change healthcare and the details and the data quality and everything that we rightfully talk about.

I think once you can actually use it and you democratize the use of something and you see how interesting and cool it can be, then the underpinnings of it which is generative AI, generating new thoughts, images, videos, whatever….Think about the future, even happening right now, you can generate many different structures, protein structures, biology. You can generate you know different chemical structures. Because at the end of the day, the English language, or whichever language it might be that ChatGPT uses, in chemistry and biology you also have a language. Whether it's the amino acids in biology and in chemistry it's the different atoms. At the end of the day, I think you're going to see an arc. It's going to take time, to Noga's point, it's going to take time because there's going to be a lot of ups and downs but that gets me excited because that's new novel insights of molecules and structures that we can create and ultimately think about. Even protocols we write, you can actually generate that if you have the right data set so you make things faster so that our clinicians or data scientists can focus on the right areas. That gets me excited because I think it's just making it a little bit more tangible for people and if done the right way, I think it can also progress a deeper understanding and appreciation for what we are all collectively trying to do in healthcare

[Evelyn] Thank you and yes it feels like there certainly hasn't been a time where there's been more dinner table conversations around AI and certainly Covid helped with that. I think everyone now understands…

[Najat] Yeah can I just say Evelyn, it's so funny my dad, who you is pretty engaged, he tells me the other day, "Hey I'm doing this project proposal for this non-profit,:- he does a lot of non-profit work-and in that one of the core focus was AI machine learning. I was like I'm so happy to see that. This is women in underprivileged areas ensuring that they get the right care early on and Telehealth and all the stuff. Then I said to him, "I'm so glad to see that." He's like, "Why, it's mainstream. Why are you surprised?" And I was like whoa that's somebody that would have not said that a year ago. That was a personal moment for me when I heard him say that.

[Evelyn] Amazing! Yeah, valuable data point for us to have as we're all in it 24/7 to get that like oh actually there's diffusion into everyone's lives at this point.

[Evelyn] I know we're at time. I would love to keep talking to our panel for  another hour or more but we are at time and I do want to just still give the opportunity for any attendees who are able to stay on the line that wanted to ask questions or hear the responses to some of the questions you have. I will pass things over to Sydney at Xtalks to help us wrap up with that session.

[Sydney] Thank you very much, Evelyn and thank you all for that very insightful presentation. We have time for maybe a question or two.I've already received a few from our audience, so I will start with those.

Our first question is “How to evaluate the method to deal with missing data from RWD?”

[Najat] I will have to drop, but I can quickly answer that question. There are many different approaches in terms of looking at missing data, for automating. First thing is definitely take a training data set and then you look through it and say what are the parameters or elements I'm looking for. There's quite a few methodologies that have been already published, for instance by Sebastian Schneeweiss at Harvard and others. There's actually an approach that we use that framework to say, okay what are the key elements, whether it's demographic or some of the things, endpoints that you're looking at for that question. Always remember the intended use and then we see what is the missing data. Then there's a threshold of how much missingness you can have. Generally we don't go with one dataset, we look at two, three different datasets. A lot of the time, what we do, it's an iteration. We work with our partners back and forth in terms of where we need to curate more, where we need to fill gaps and in parallel we're also having conversations with regulators just to say, okay we have ORR but progression-free survival, there's this XYZ is lacking.

So, it's not like a static approach and I think that's the beauty of it. With almost anything in science, it's pretty iterative but that's what I would say. It’s a lot of what we're doing, there are frameworks and methodologies. I don't think there's one that everybody uses,  you can also use different ones, specifically for oncology versus some of the other disease areas.  I'm sure Noga and Jesper will have many more great insights. Great to see everyone! I will have to drop. Thank you so much for having me, again!

[Evelyn] Thanks, Najat! Thanks for coming. Anything to add from Noga or Jesper?

[Jesper] A quick one from the regulatory perspective. I appreciate the references to Sebastian's name and others and assimilation of how to do that. That's definitely a useful approach to consider. I would just like to point to the fact of the Data Quality Framework we have released where the implementation of that towards real-world data is something we'll do as a demonstration project over the next half year. So, maybe a space to watch to better understand the regulatory perspective around data quality and missing data. Something to watch out for.

[Noga] And I'll just say very briefly that I think because we work directly with the patient, our approach here is pretty different. We actually make an effort to look at the individual patient and, using claims data, understand every interaction they've actually had in the healthcare system. Then we can literally go chase those down one by one and actually ensure that we have that really complete-I think people are calling it "past complete" picture. You're not going to do that on a national scale or continental scale I should say, but I think when you're working with smaller patient-centered cohorts it is actually possible to chase that information down and get to completeness, at least in set windows of time that you can define.

[Sydney] All right thank you all for those answers. We'll get to one last question. So, earlier the panel mentioned the importance of representative RWD. This audience member would like to know if any of the panelists could speak to promising ways that they are seeing RWD being used to study diverse, often underrepresented patient groups and sub-populations.

[Noga] Yeah, I can speak to that. I think…oh man I hate to say this….but as I mentioned earlier in another context that the bar is so low and I think this is another place where we see things like, at PicnicHealth, the cohorts that we build, we have this very deep patient-centered data compared to running an observational study that you have to recruit into or compared to a clinical trial population. You are asking some effort from patients to sort of sign up and make a contribution but it's a five minute sign up that patients can do from their home. So you're not getting that same barrier that you would see for example in clinical trial participation. I think when we see those representative, more diverse datasets, that the reality is that sometimes I think, for example our work in MS,  it's just kind of shocking how little people understand about what the disease even looks like in non-white populations. We know black women are more affected than other populations. We know their disease looks really different, has different outcomes, but I think we're just starting to piece together what that even looks like. For better or worse, I think the reality is that sometimes datasets that give access to a look at diverse populations end up just being about the baseline of getting to the core understanding you have about a disease for a broader population and making sure that extends across the reality of the whole population.

[Sydney] All right well thank you so very much for those answers and for this presentation. We've reached the end of the Q&A portion of the webinar. If we couldn't attend to your questions, the team at PicnicHealth may follow up with you or if you have any further questions you can direct them to the email address on your screen. So thank you everyone for participating in today's webinar. You'll be receiving a follow-up email from Xtalks with access to the recorded archive for this event. The survey window will be popping up on your screen and your participation is appreciated as it will help us to improve our webinars.

Now please join us in thanking our speakers Evelyn Pyper, Noga Leviner, Najat Khan and Jesper Kjaer. We hope you found this webinar informative. Have a great day, everyone.