Tx-LLM: Supporting therapeutic development with large language models

Most candidates for therapeutic drugs fail clinical trials. Even if successful, they typically require 10–15 years and $1–2 billion to develop. A major reason for this is that the development pipeline contains many steps and many independent criteria that a therapeutic must satisfy. For example, a therapeutic should interact with its specific target but not other entities, producing the desired functional improvement but not off-target toxicity. Additionally, it should be able to travel to its desired destination, be cleared out of the body in an appropriate amount of time, and be suitable for manufacturing at scale. As one can imagine, measuring these properties experimentally is expensive and takes a long time, leaving an opportunity for an alternative approach: using machine learning (ML) to predict these properties quickly and efficiently.

To that end, we introduce Tx-LLM, a large language model (LLM) fine-tuned from PaLM-2, to predict properties of many entities (e.g., small molecules, proteins, nucleic acids, cell lines, diseases) that are relevant to therapeutic development. Tx-LLM is trained on 66 drug discovery datasets ranging from early-stage target gene identification to late-stage clinical trial approval, and therefore, is best suited to research on therapeutic applications. With a single set of weights, Tx-LLM achieved competitive performance with state-of-the-art models on 43 out of the 66 tasks and exceeded them on 22. Interestingly, we also observed that Tx-LLM exhibited abilities to combine molecular information with textual information as well as to transfer capabilities between tasks with diverse types of therapeutics. Overall, Tx-LLM is a single model that may be useful throughout the development of therapeutic drugs pipeline.

Curating the Therapeutics Instruction Tuning (TxT) collection for fine-tuning LLMs

Gathering data throughout the development pipeline is key to training the Tx-LLM model. We leveraged data from the Therapeutic Data Commons (TDC), a public collection of drug discovery datasets for training ML models, and processed 66 tasks most relevant to drug discovery into instruction-answer formats suitable for LLMs. Our collection, referred to as Therapeutics Instruction Tuning (TxT), structures each prompt with instructions, a context, a question, and an answer. Few-shot exemplars were also included in the question to enable in-context learning. TxT tasks can be broken down into three types:

• classification, which is framed as a multiple-choice question (e.g., output whether a drug is [A] non-toxic or [B] toxic)
• regression (e.g., output the binding affinity of a drug to a protein)
• generation (e.g., output which molecules were used in a chemical reaction)

TxT extends beyond the data found in TDC in a few notable ways. The context in each prompt supplies additional information that allows a generalist model to distinguish subtasks (e.g., training the model to predict toxicity in multiple different assays). Furthermore, some features in our dataset, such as cell lines, are represented directly as text as opposed to mathematical objects, such as gene expression vectors, where each entry of the vector is how much the cell expresses a particular gene. As we’ll see later, this representation also allows Tx-LLM to leverage its pre-training on natural language.

Training Tx-LLM on TxT can outperform state-of-the-art specialist models

We then evaluated Tx-LLM on TDC datasets and compared it with existing state-of-the-art specialist models. Tx-LLM achieved competitive performance on 43 out of 66 tasks and exceeded their performance on 22. In many cases, we also observed that Tx-LLM was effective at predicting numeric values, which was somewhat surprising since LLMs have previously been observed to struggle with mathematical tasks. Binning the predictions into integers between 0 and 1000 may have helped with this, as it keeps the prediction format consistent and independent of units.

One area in which Tx-LLM was particularly effective was the combination of small molecules and text (e.g., given a drug and a disease name in a clinical trial, predict whether the drug would be approved). Tx-LLM actually outperformed current top models for the large majority of tasks involving these small molecules. This is likely because Tx-LLM was pre-trained on text including information about various diseases, so some context was already encoded in its weights.