← All workThesis

Equivariant diffusion for 3D molecules

My bachelor's thesis: a diffusion model that generates 3D molecules, with PaiNN inside, built from scratch in PyTorch.

Role
Bachelor's thesis, DTU
Year
2025
Stack
PyTorch, Diffusion models, E(3) equivariance, PaiNN, QM9
Loading trajectories
1 of 8
HydrogenCarbonNitrogenOxygenFluorine
Real sampling runs of the trained model (9 PaiNN layers, 256 features, cosine noise schedule, EMA weights), shown at every second of its 1000 denoising steps. Each atom takes the colour of the element the model would assign it at that step, and the colour strengthens as the noise falls away. Bonds follow the distance rules of the stability check. In this run of 64, 55 molecules came out stable (99% of atoms); these 8 are picked from those.

A molecule is a handful of atoms in 3D space. Know where each one sits and what element it is, and you know the molecule. A diffusion model for molecules learns to produce exactly that: it starts from a cloud of random points with random identities, and over a thousand small steps nudges them into something a chemist would recognise. The figure above is the model I trained, doing just that.

For my bachelor’s thesis I rebuilt Equivariant Diffusion Models (Hoogeboom, Garcia Satorras, Vignac and Welling, 2022) from scratch in PyTorch, with PaiNN (Schütt, Unke and Gastegger, 2021) as the network inside, and trained it on QM9, a dataset of about 134,000 small organic molecules. It was a learning project as much as a reproduction, so I wrote all of it myself: the diffusion process, the equivariant network, training, sampling and the benchmarks. I did it in spring 2025, supervised by Mikkel N. Schmidt at DTU Compute.

Symmetry, built in

A molecule has no preferred orientation. Turn it around and it is the same molecule, so a model that generates molecules shouldn’t care which way they face. You can teach that by showing every molecule from many angles, or you can build it into the network so it holds exactly. EDM does the second. The network is equivariant: rotate its input and its output rotates the same way. Position is handled by keeping every molecule’s centre of mass at the origin, so there is nothing to learn about where a molecule sits either.

Start from noise with no preferred direction, take equivariant steps, and the whole sampler is equivariant: turn the starting noise, and the model has to produce the same molecule, turned. Here it is on the trained model.

Noise, t = 1000
Molecule, t = 0
The same noise, turned 146°
What the model made of it
The bottom right is not a rotated copy of the top right: it is what the sampler actually produced when every bit of positional noise was turned by 146°, seen down the rotation's own axis. Over all 64 molecules of the run and all 1,000 steps, every atom lands within a thousandth of an ångström of the rotated original, which is floating-point rounding, and every molecule ends up with the same elements.
Why it has to: the proof

One reverse step of the sampler updates the atom positions as

zt−1=c1 zt+c2 ϵ^(zt,t)+c3 ϵt,z_{t-1} = c_1 \, z_t + c_2 \, \hat{\epsilon}(z_t, t) + c_3 \, \epsilon_t ,

where ϵ^\hat{\epsilon} is the network’s noise prediction, ϵt\epsilon_t is fresh Gaussian noise, everything is centred on the centre of mass, and c1,c2,c3c_1, c_2, c_3 are numbers that depend only on tt.

Take any rotation RR, and suppose a step starts from RztR z_t and draws RϵtR \epsilon_t. The network is equivariant, so ϵ^(Rzt,t)=R ϵ^(zt,t)\hat{\epsilon}(R z_t, t) = R \, \hat{\epsilon}(z_t, t), and

c1Rzt+c2R ϵ^(zt,t)+c3Rϵt=R zt−1.c_1 R z_t + c_2 R \, \hat{\epsilon}(z_t, t) + c_3 R \epsilon_t = R \, z_{t-1} .

Centring doesn’t get in the way: the centre of mass of RzR z is RR times the centre of mass of zz. The atom types ride along unchanged, because the network’s type output is invariant. So if a run starts from RzTR z_T and every noise draw is rotated, every step stays exactly rotated, all the way to z0z_0. That is the figure above.

A standard Gaussian looks the same from every direction, so RϵR \epsilon is exactly as likely as ϵ\epsilon. Every rotated run is as likely as the original, which means the model gives a molecule and any rotation of it the same probability:

p(Rz0)=p(z0).p(R z_0) = p(z_0) .

This is the argument of Xu et al. (2022), which EDM builds on.

PaiNN first

PaiNN is a message-passing network. Atoms send each other messages along the edges between them, and each atom carries two kinds of features: plain numbers that stay put when the molecule turns, and 3D vectors that turn with it. That split is what makes the whole network equivariant.

Drag the molecule to turn it

Rings: unchanged. Arrows: turned 0°.

Scalar part: built from distances, so turning the molecule leaves it aloneVector part: built from directions, so it turns with the molecule
One round of message passing, sketched on the imidazole from the cover. An atom hears from every atom within PaiNN's 5 Å cutoff, weighted by its smooth cutoff function: the ring sums those weights, the arrow sums the weighted directions. The learned filters of the real network are left out. Drag to turn it, click an atom to follow it.

Before trusting my implementation inside a diffusion model, I trained it as a property predictor on QM9, untuned, for ten hours per property.

PropertyUnitPaiNN paperMy PaiNN
Dipole momentD0.0120.014
Polarisabilitybohr³0.0450.082
Zero-point energymeV1.281.27
Internal energy at 0 KmeV5.857.80
Internal energy at 298 KmeV5.838.04
EnthalpymeV5.987.82
Free energymeV7.358.65
Heat capacitycal/mol K0.0240.025
Mean absolute error on the QM9 test set, lower is better. Paper numbers from Schütt et al. (2021).

Close to the paper everywhere except polarisability, with no tuning at all. Getting there took two detours I still remember. The dipole moment head ends in the length of a vector, so it can only predict positive numbers, and I had standardised the targets, which put half of them below zero. It never went below about 2.5 debye until I switched to min-max scaling. For the energies, predicting what is left after subtracting each atom’s tabulated reference energy, rather than the total, took the error from about 2,000 meV to 7.8.

Diffusion on molecules

EDM noises the atom positions and the atom types together. Positions are 3D coordinates. Types are one-hot vectors over hydrogen, carbon, nitrogen, oxygen and fluorine, scaled down by a factor of 0.25. At each of the 1,000 reverse steps, PaiNN looks at the noisy molecule and predicts the noise in both: a vector per atom for the positions, and five numbers per atom for the type.

[ϵ^(x),ϵ^(h)]=ϕ(zt(x),zt(h),t)\left[\hat{\epsilon}^{(x)}, \hat{\epsilon}^{(h)}\right] = \phi\left(z_t^{(x)}, z_t^{(h)}, t\right)

The position part has to rotate with the molecule and the type part must not, which maps neatly onto PaiNN’s vector and scalar features. My final model has 9 PaiNN layers with 256 features, a cosine noise schedule, and samples with an exponential moving average of its weights.

Making it train was the hard part. The noise sometimes puts two atoms almost on top of each other, which made PaiNN’s distance features blow up, and late in sampling the cosine schedule’s coefficients grow large enough to fling a trajectory off to infinity. Small, deliberate fixes handled both.

The type scaling turned out to matter more than anything else I tuned: without it, molecule stability fell from 87% to 58%. Watching trajectories showed why. With unscaled types the model commits to its elements early and then refuses to change them, even when the geometry around them is wrong. With scaled types it keeps reconsidering until the last hundred steps or so, and you can see that in the figure above, where the colours keep flickering right up to the end.

Results

ModelAtoms stableMolecules stableValidValid and unique
QM9 itself99.095.297.797.7
Mine, cosine schedule98.887.194.693.0
Mine, linear schedule98.380.892.290.8
EDM (Hoogeboom et al., 2022)98.782.091.990.7
END (Cornet et al., 2024)98.989.194.892.6
Percentages over 10,000 generated molecules, higher is better. A molecule is stable when every atom has the right number of bonds; validity and uniqueness are checked with RDKit. Other models' numbers are from their papers.

The from-scratch model edges past the original EDM on every metric, though newer models such as END, the paper the project first set out to replicate, go further. The cosine schedule beats the linear one by six points of molecule stability. It spends more of its steps at low noise, where the final details are placed, which works like a slow-motion finish. Running the model again for this page, 55 of 64 fresh samples came out stable, in line with the 87% above.

Missing edges are information

The first full-size runs beat the original paper by a wide margin: 99.3% of atoms and 91.6% of molecules stable, against EDM’s 98.7% and 82.0%. Reproductions don’t usually go that way, so I went looking for the mistake. My stability check gave the same numbers as the EDM authors’ own code when both were run on QM9 itself, so the benchmark was sound.

The clue was diversity. On a larger sample only about 40% of the molecules were unique, and the model favoured compact shapes over the stretched-out ones that are common in QM9. The cause was a leftover from the property-prediction work: preprocessing dropped every edge between atoms more than 5 Å apart.

10 edges from this atom20 edges from this atom, 10 of them new
5 Å
One of the longer molecules from the run, about 10 Å end to end, seen from the atom at one end. In training, its edges stopped at 5 Å, so an atom without an edge was far away. When sampling, every atom had an edge, so to the model every atom looked close. Across the whole molecule, 84 of its 210 atom pairs are more than 5 Å apart.

So the model had learned that a missing edge means “far away”. At sampling time every pair was connected, so it read every molecule as one where everything is close together, and built compact molecules: easier to get right, but often the same ones. Fixing the preprocessing gave the clean numbers above. The more interesting part was what the bug had shown: the model reads the graph’s edges as a description of the molecule’s shape, and it uses them.

So I used them on purpose. Train on graphs that keep only each molecule’s shortest edges, and at sampling time start from the edge pattern of a real molecule of the right size instead of a fully connected graph. A sweep over how many edges to keep, on smaller models:

Molecules stableValid and unique
50%60%70%80%90%100%90%80%70%60%50%40%30%20%86.2%65.6%
Edges kept in training
Smaller models (5 layers, trained on a fifth of QM9) with every edge kept, then fewer and fewer: only the shortest edges of each molecule are kept, and sampling starts from the edge pattern of a real molecule of the same size. 2,000 molecules per model; hover for the numbers.

Taking edges away first hurts, then helps a lot. With 40% of the edges, a 5-layer model reaches 86% molecule stability, close to the full 9-layer model with every edge, and with less than half the message passing it trains and samples faster too. At full size:

Full-size modelAtoms stableMolecules stableValidValid and unique
Every edge98.887.394.192.8
40% of edges, real edge patterns99.191.594.365.3
Both trained with an atom-type scaling of 0.2; 10,000 molecules each. Thesis Table 6.4.

91.5% of molecules stable is more than any model in the results table, END included, and more than GeoBFN, the strongest model in my thesis’s comparison at 90.9%. The cost is diversity: the edge pattern settles the rough shape, so far fewer of the molecules are unique.

It is conditional generation, not generating from scratch: the edge patterns come from real molecules. But it points at something useful. Saying which atoms sit within a few ångström of each other is a far easier problem than placing every atom in 3D. Predict that rough sketch, with a simpler model for instance, hand it over, and this model completes the geometry, more reliably than it does from nothing. With the atom types given as well, it starts to look like a fast first guess at a molecule’s structure, the kind of thing expensive quantum chemistry is used to refine.

What I learned

Most of the work in a reproduction is making sure you are measuring the right thing. The result I’m happiest with started as a number I didn’t believe: chasing it down turned a preprocessing leftover into a way to steer the model. The thesis also gave me a working understanding of diffusion models, from the discrete DDPM view to the continuous SDE one, and of how symmetry can be built into a network instead of learned. One stretch goal went unfinished, a learnable forward process, and I picked it up again the following semester in two-way neural flow bridges.

Next projectPolyPred →