Reinforcement Learning with learnt models for molecular design

Description

The design of novel molecules with specific properties is a central challenge in fields such as drug discovery and materials science. This project aims to develop methods to integrate reinforcement learning (RL) with molecular simulation to facilitate automated molecule design and optimization. 

Reinforcement learning allows agents to learn optimal decision-making strategies by interacting with environments and receiving feedback. When coupled with molecular simulation techniques such as molecular dynamics (MD), RL can be employed to explore the vast chemical space efficiently and generate molecules with desirable properties, such as high binding affinity, stability, or reactivity. The proposed research will focus on developing a framework where an RL agent interacts with a molecular simulation environment to propose candidate molecules, which are then evaluated based on molecular dynamics simulations. The agent will be required to handle both combinatorial search spaces and multi-objective optimization targets.

A key limitation of the above approach is the computational cost involved in running molecular simulations to evaluate the generated molecules. Computational chemistry simulations are known to be very compute intensive, often taking hours to evaluate a single molecule. This has led to a number of approaches that attempt to use cheaper scoring functions to evaluate generated molecules. Unfortunately these scoring functions often provide inaccurate evaluations, leading to suboptimal results.

This project aims to replace computationally expensive simulators in the molecule optimization process with learnt models. These models must provide accurate evaluations at significantly reduced computational costs. A key challenge is ensuring that the emulators' uncertainty is appropriately reflected and incorporated into the optimization framework, to mitigate issues of generalization that may arise during optimization. This PhD research will involve several key steps:

1. Design of a Reinforcement Learning (RL) Framework for Molecule Optimization: The project will focus on developing RL methods capable of navigating the combinatorial space of molecular design. Specifically, we will investigate approaches such as Monte Carlo Tree Search (MCTS) and other techniques suited for handling large, complex search spaces.

2. Development of Learning-based Emulators and Uncertainty Quantification: Methods for training emulators that can replace traditional simulators in the optimization loop will be developed. A critical component will be the ability to quantify and incorporate uncertainty in the emulator's predictions to ensure reliable decision-making.

3. Evaluation on Drug Design Use Cases: The framework will be tested on practical use cases in drug design, with potential extensions to account for domain-specific secondary objectives (e.g., pharmacokinetic properties or safety profiles) that must be satisfied by candidate molecules.

References:

1. Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation https://arxiv.org/abs/2110.01219

2. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model https://arxiv.org/abs/1911.08265

This project will be a collaboration between the IMEC AI & Algorithms department and the AI-lab at the Vrije Universiteit Brussel (VUB).

The AI&Algorithms (AI&A) department at IMEC is known for its expertise in AI and algorithms at the intersection of SW and HW. With a team of 250 researchers,the department specializes in edge AI, modeling of sensors and sensor fusion, signal processing, development of learning algorithms and high-performance computing (HPC). A large part of the research group works on generic technological building blocks transferrable to multiple domains. However, the group recognizes the crucial role of healthcare and prevention in society and aims to leverage its expertise to address major health challenges.

VUB’s Artificial Intelligence Lab (​ai.vub.ac.be​) was founded in 1983 by Luc Steels and was the first AI Lab in Europe. Under the leadership of Prof. Ann Nowé, it counts 60 researchers including 11 professors. The lab has a publication record of more than 850 publications at major conferences and journals, and is well embedded in the international research community. It has always had a strong focus on embodied AI, and has a solid track record in fundamental as well as applied research, both on the European level (9 EU projects in the last 5 years) and the national level. The lab has a long-standing history of research on reinforcement learning, multi-agent systems, and natural language and speech processing. Furthermore, the lab was recently enriched with researchers from the fields of computational creativity, human-AI dynamics, knowledge representation and reasoning, and constraint optimization.