A collaborative study led by Professor Muneki Ito from the University of Tokyo has revealed that non-equilibrium thermodynamics provides a theoretical explanation for the application of optimal transport theory in generative models. Non-equilibrium thermodynamics, a branch of physics that focuses on dynamically changing systems, while optimal transport theory is a mathematical tool for optimizing distribution changes to minimize cost. This discovery introduces a novel thermodynamic approach to the field of machine learning. The related results have been published in the journal Physical Review X.
Currently, image generation technology is advancing rapidly, driven by diffusion models that are characterized by the introduction of randomness through "noise." During training, noise is added to the original data via diffusion dynamics. During generation, the model removes the noise to produce new content. The key lies in precisely controlling when and how much noise to add, which directly affects the quality of the generated content.
Muneki Ito pointed out: "The choice of diffusion dynamics, or the noise schedule, has long been controversial. Although empirical evidence shows that optimal transport dynamics perform well in diffusion models, its theoretical basis remains unclear." The research team utilized thermodynamic trade-off relations — a technique describing the relationship between thermodynamic dissipation and the speed of system changes — to derive an inequality between thermodynamic dissipation and the robustness of data generation in diffusion models. This inequality proves that optimal transport dynamics can ensure the most robust data generation.
"The results show that for real-world image generation scenarios, our bounds fit closely within a certain order of magnitude," Ito said. "This indicates that the inequality not only helps understand the optimal strategy in diffusion models but also has guiding significance for analyzing the practical application of generated image data." Notably, part of the research was carried out by undergraduates, especially first author Kotaro Ikeda, who made outstanding contributions in numerical calculations and theoretical analysis.
Ito stated: "We hope the research results will raise awareness in the machine learning community about the importance of non-equilibrium thermodynamics and inspire researchers, including the next generation, to continue exploring its practical applications in biological and artificial information processing."
