A joint research team led by Professor Muneki Ito from the University of Tokyo has achieved a significant breakthrough. The related results were published in the journal Physical Review X. Using non-equilibrium thermodynamics (a branch of physics that studies continuously changing systems), the study reveals why optimal transport theory (a mathematical framework for optimizing distribution changes to minimize cost) enables generative models to reach optimality. Given that non-equilibrium thermodynamics has not been fully utilized in generative model design before, this discovery opens a novel thermodynamic pathway for machine learning research.
In recent years, image generation technology has developed rapidly, and what were once top-tier videos of celebrities eating spaghetti are now considered mediocre. The algorithms driving image generation are diffusion models, which incorporate a random element called "noise." During training, noise is introduced into the original data through diffusion dynamics. During generation, the model must remove the noise to create new content from noisy data, which can be achieved by considering time-reversed dynamics (similar to playing a video backward). One of the keys to building high-quality content generation models is determining when and how much noise to add to the data.
Lead researcher Professor Muneki Ito pointed out that the choice of diffusion dynamics (i.e., the noise schedule) has long been controversial. Although empirical evidence suggests that optimal transport dynamics are useful in diffusion models, there has been a lack of theoretical proof. Although diffusion models were initially inspired by non-equilibrium thermodynamics, and optimal transport theory is closely related to this field, previous studies overlooked this connection. Thus, the research team raised the question: Can non-equilibrium thermodynamics provide a theoretical framework for the effectiveness of optimal transport dynamics in diffusion models?
Recent advances in thermodynamic trade-off relations (techniques describing the relationship between thermodynamic dissipation and the speed of system changes) provided support for this. The researchers used this technique to derive an inequality between thermodynamic dissipation and the robustness of data generation in diffusion models, thereby proving that optimal transport dynamics can ensure the most robust data generation. Ito explained that for real-world image generation scenarios, the newly derived inequality is tightly relevant within a certain order of magnitude. This not only helps understand the optimal protocol in diffusion models but also has significant practical implications for analyzing generated image data.
Notably, this project also had an unexpected highlight. Both the first author Kotaro Ikeda and the second author are undergraduates, and part of the research was conducted as part of their coursework. Kotaro Ikeda made significant contributions to the study, ranging from numerical calculations to theoretical analysis.
Professor Ito expressed hope that the research results will raise awareness in the machine learning community about the importance of non-equilibrium thermodynamics. In the future, researchers, including the next generation, will continue to explore its practical applications in understanding biological and artificial information processing.
