Training a residual neural network with L2 regularization on weights and biases is equivalent to minim- izing a discrete least action principle and to controlling a discrete Hamiltonian system representing the propagation of input data across layers. The kernel/feature map analysis of this Hamiltonian system suggests a mean-field limit for trained weights and biases as the number of data points goes to infinity. The purpose of this paper is to investigate this mean-field limit and illustrate its existence through numerical experiments and analysis (for simple kernels).
Mean-field limits of trained weights in deep learning: A dynamical systems perspective
Smirnov A., Hamzi B., Owhadi H. (2022) "Mean-field limits of trained weights in deep learning: A dynamical systems perspective " Dolomites Research Notes on Approximation, 15(3), 125-145. DOI: 10.14658/PUPJ-DRNA-2022-3-12
Year of Publication
Dolomites Research Notes on Approximation
Serial Article Number