Artificial intelligence (AI) and machine learning (ML) are pretty much everywhere there are computers and connected devices. Targeted advertising relies on AI. So do customer-service chatbots on websites, predictive typing in internet search engines and email apps, and virtual assistants like Siri and Alexa.
The field took a giant leap forward in the public’s consciousness in late 2022 when OpenAI released ChatGPT, a generative AI engine that can produce detailed text and images with just a few human prompts. Other organizations, including a few tech giants, have since followed suit.
While some have used ChatGPT and the like for entertainment purposes, AI and ML are starting to find a home in life sciences, including in support of clinical trials.
In May 2023, the US Food and Drug Administration (FDA) released a “discussion paper” on AI and ML in drug development to stimulate dialog among industry, academic and regulatory stakeholders. The agency noted that it had “seen a significant increase in the number of drug and biologic application submissions using AI/ML components over the past few years”, including more than 100 in 2021 alone.
Similarly, the European Medicines Agency (EMA) published a draft “reflection paper” in July 2023 about applying AI to drug development and post-market surveillance. The European agency announced closing public comments on the draft on December 31, 2023, at the time of writing this document.
The EMA recommended that researchers consider the complexity of their trial and how AI/ML will be used to conduct benefit‒risk assessments.
According to the EMA document: “The application of AI in the process of drug discovery may be a low-risk setting from a regulatory perspective, as the risk of non-optimal performance often mainly affects the sponsor. However, if results contribute to the total body of evidence presented for regulatory review, principles for non-clinical development should be followed.”
The EMA said that AI might be useful for drafting, compiling, translating and reviewing data that may belong in product labels. Following marketing authorization, the technology “can effectively support, for example, pharmacovigilance activities including adverse event report management and signal detection.”
AI could be a powerful assistant for clinical trials development and execution
At the 2022 New England Journal of Medicine Artificial Intelligence in Medicine Symposium (AIMS), Adam Dunn, head of biomedical informatics and digital health at the University of Sydney School of Medical Sciences, spoke about how AI and ML can improve the design, reporting and synthesis of clinical trials.
He suggested that AI and ML can play a key role in matching patients to trials by searching unstructured data from electronic health records (EHRs). Natural-language processing (NLP) can fill in gaps in traditional clinical studies via EHR information extraction, document classification and inference from multimodal data.
“Machine learning is going to play an increasingly important role in finding patients and filling in information that is not recorded in EHRs,” Dunn said. NLP technology extracts unstructured data “buried” in clinical notes and harmonizes data pulled from varying locations.
“Replacing control arms would substantially reduce the cost of impactive trials and make trials possible where recruitment was historically a critical barrier to completing,” Dunn explained.
“The gap between randomized control trials and retrospective cohort analyses from EHRs is going to get increasingly blurry and machine learning will play a key role in how we learn to replace parts of individual trials with routinely collected data from many patients,” he added.
What are the issues and concerns with using AI/ML in life sciences?
AI development is not without growing pains, as evidenced by the sacking and subsequent re-hiring of OpenAI co-founder Sam Altman in November 2023. The uncertainty is evident in life sciences as well.
In an article published in the journal Health and Technology in February 2023, researchers from the Massachusetts College of Pharmacy and Health Sciences (MCPHS) said that AI in clinical trials is “in its relative infancy.”
The MCPHS investigators reviewed 48 academic papers and nine regulatory documents and concluded that clinical trial-generated medical evidence is “likely to remain the gold standard for development of safe and effective drugs, despite the long-standing acknowledgment of the great investment and high risks involved for pharmaceutical companies.”
A recent preprint study from researchers at Stanford University and the University of California, Berkeley, found that OpenAI’s GPT-4 large language model became less accurate between March and June 2023, while its supposedly less advanced predecessor, GPT-3.5, improved.
Dunn recalled that he had a discussion with colleagues a decade ago about how best to organize systematic reviews. “We landed on this idea of constructing systematic reviews … that could watch how a human performed the individual steps … and automatically repeat each of those steps over and over and over again at the press of a button,” he said.
These ideas were more practical than novel in 2013, as AI was not able to generate such clinical trial data available then, according to Dunn.
While ML today can help investigators quickly synthesize results from multiple trials, problems persist.
“Some of these methods are quite useful, but many of them include flawed assumptions about the quality of the data that they’re modeling,” Dunn said. “I would urge caution here because there seems to be a disconnect between the computer science advances where method novelty is much more important than the data quality and how that method might be deployed and applied research in clinical epidemiology, where the opposite is true.”
A big issue is how to unlock individual participant data (IPD) to use as synthetic control or comparative arms in future research. “We can simply turn our idea of what a [randomized control trial] is upside-down and run lots of single-arm studies against the aggregate of IPD from all prior trials,” Dunn said.
Dunn and colleagues have been developing and testing methods for assigning trials for systematic reviews and safety meta-analyses by coupling information from trial registrations like the US government’s clinicaltrials.gov and articles listed in PubMed.
“The aim is to rank trials from their registrations or published articles against a systematic review,” he explained. “We take a systematic review protocol and we try and identify which trials belong to that systematic review. Or if we have a systematic review that’s been published and it has already included trials in it, we are looking for new trials that should be added to a systematic review if it was updated.”
Still, Dunn said that experts need to supervise these automated systematic reviews.
“Currently, the best tools we can hope for in this space are likely to be ones that monitor for new registrations and trial results … and then add extra annotation and other information extraction tools to help systematic reviewers quickly update meta-analyses,” Dunn said.
The EMA agreed in its draft reflection paper: “AI applications used for drafting, compiling, translating, or reviewing medicinal product information documents are expected to be used under close human supervision.”
“Given that generative language models are prone to include plausible but erroneous output, quality review mechanisms need to be in place to ensure that all model-generated text is both factually and syntactically correct before submission for regulatory review.”