Estimating model uncertainty of neural networks in sparse information form

We present a sparse representation of model uncer_tainty for Deep Neural Networks (DNNs) where the parameter posterior is approximated with an inverse formulation of the Multivariate Normal Distribution (MND), also known as the informa_tion form. The key insight of our work is that the information matrix, i.e. the inverse of the co_variance matrix tends to be sparse in its spectrum. Therefore, dimensionality reduction techniques such as low rank approximations (LRA) can be effectively exploited. To achieve this, we develop a novel sparsification algorithm and derive a cost_effective analytical sampler. As a result, we show that the information form can be scalably applied to represent model uncertainty in DNNs. Our exhaustive theoretical analysis and empirical eval_uations on various benchmarks show the competi_tiveness of our approach over the current methods.