TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision

The Big Picture

Imagine trying to predict the weather by snapping photographs every few minutes and guessing what comes next, but each guess inherits all the error from every previous guess, until your forecast is nothing but noise. That is the challenge facing scientists who use neural networks to solve partial differential equations (PDEs). Small inaccuracies compound over time, and what starts as a promising simulation drifts away from reality.

PDEs are the mathematical backbone of modern science. From the fluid dynamics of jet engines to the quantum evolution of exotic materials, they describe how things change in space and time. Traditional numerical solvers have served well for decades, but they struggle when problems get too complex or involve too many interacting variables. Neural networks offer an alternative: flexible tools that can handle complexity traditional methods choke on. The catch? Accuracy, especially over long time horizons, has been hard to achieve.

A team of researchers at MIT and Harvard, affiliated with the NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI), set out to fix that. Their method, Time-Evolving Natural Gradient (TENG), doesn’t just improve on existing neural PDE solvers. It hits machine precision, the limit of what a computer can represent numerically.

Key Insight: TENG combines ideas from quantum physics, differential geometry, and optimization theory to solve PDEs with neural networks at a level of accuracy that rivals traditional high-precision numerical methods, a first for this class of approaches.

How It Works

To understand TENG, it helps to know the two older approaches it builds on. The first, time-dependent variational principle (TDVP), projects the PDE’s dynamics directly onto the geometry of the neural network. Think of it like casting a shadow: take the “true” direction the solution should evolve, then find the nearest direction the network can actually move. The second, optimization-based time integration (OBTI), treats each time step as a fresh optimization problem, finding the network parameters that best match where the solution should be after a small step forward.

Both approaches hit a ceiling on accuracy. TDVP accumulates projection errors; OBTI can settle into imprecise solutions. TENG asks: what if you combined the geometric intuition of TDVP with the optimization power of OBTI, and then used natural gradient methods?

Natural gradient, originally developed by Shun-ichi Amari in 1998, is a smarter way of adjusting a neural network’s internal settings. Standard optimization takes the shortest apparent path toward a better solution, but that path ignores the actual shape of the loss surface, sometimes veering into unhelpful valleys. Natural gradient accounts for how the network’s outputs actually respond to changes in its parameters, producing more informed steps. This matters most when the loss surface is highly curved, which is exactly what happens when fitting a neural network to a rapidly changing PDE solution.

Here’s how TENG works, step by step:

1. Initialize: Fit the neural network to the initial condition of the PDE.
2. Project: At each time step, define a loss function in the space of functions (not parameters), measuring how well the current network matches where the solution should be.
3. Optimize with natural gradient: Iteratively update the network parameters using natural gradient steps, nudging the network toward the desired change, constrained to what the network can actually express.
4. Advance: Move forward in time and repeat.

The team developed two algorithms from this framework. TENG-Euler is the basic first-order variant, fast and already far more accurate than competing methods. TENG-Heun goes further: it evaluates the gradient at intermediate steps, much like how Runge-Kutta methods take carefully placed intermediate measurements before committing to a full step. To keep things tractable for large networks, the authors also introduced a sparse update strategy that targets only the most informative parameters at each step, significantly reducing computational cost.

Why It Matters

Across three benchmark PDEs (the heat equation, the Allen-Cahn equation, and Burgers’ equation), TENG achieves errors at or near the limits of floating-point arithmetic. Compared to OBTI and PINN with energy natural gradient, TENG reduces errors by orders of magnitude. This isn’t a marginal improvement; it’s a qualitative leap into a new accuracy regime.

For physics, high-precision neural PDE solvers make it possible to study complex quantum systems, turbulent flows, and other phenomena where both accuracy and scalability matter. Traditional solvers often scale poorly to high dimensions. Neural networks handle that more gracefully.

For AI, TENG shows that combining physics (variational principles, information geometry) with machine learning can produce capabilities that neither field reaches on its own. The natural gradient connection, originally developed for statistical learning, turns out to be exactly the right mathematical tool for navigating the function-space geometry of time-evolving PDEs.

Open questions remain. Can TENG scale to the high-dimensional PDEs that arise in quantum many-body physics or climate modeling? How does it handle PDEs with discontinuous solutions or sharp fronts, where accuracy is hardest to maintain? And can the sparse update strategies be pushed further, making TENG competitive on problems where even current neural solvers are too expensive?

Bottom Line: TENG achieves machine-precision accuracy in neural network PDE solving by fusing natural gradient optimization with sequential time integration, outperforming all current state-of-the-art methods by orders of magnitude.