Mean-field limits of trained weights in deep learning: A dynamical systems perspective

Training a residual neural network with L2 regularization on weights and biases is equivalent to minim- izing a discrete least action principle and to controlling a discrete Hamiltonian system representing the propagation of input data across layers. The kernel/feature map analysis of this Hamiltonian system suggests a mean-field limit for trained weights and biases as the number of data points goes to infinity. The purpose of this paper is to investigate this mean-field limit and illustrate its existence through numerical experiments and analysis (for simple kernels).