Kernel shape renormalization explains output-output correlations in finite Bayesian one-hidden-layer networks

Abstract

Finite-width one hidden layer networks with multiple neurons in the readout layer display nontrivial output-output correlations that vanish in the lazy-training infinite-width limit. In this manuscript we leverage recent progress in the proportional limit of Bayesian deep learning (that is the limit where the size of the training set and the width of the hidden layers are taken to infinity keeping their ratio finite) to rationalize this empirical evidence. In particular, we show that output-output correlations in finite fully connected networks are taken into account by a kernel shape renormalization of the infinite-width NNGP kernel, which naturally arises in the proportional limit. We perform accurate numerical experiments both to assess the predictive power of the Bayesian framework in terms of generalization, and to quantify output-output correlations in finite-width networks.

Lorenzo Giambagli

PostDoc Department of Physics, Freie Universität Berlin

My research interests include Spectral analysis of Deep Neural Network (DNN), Structura Pruning, Bayesian Inference in DNN, Simplicial Complexes Dynamics, Theoretical Neuroscience