BROWSE

Related Scientist

's photo.

수리및계산과학연구단
more info

ITEM VIEW & DOWNLOAD

α-stable convergence of heavy-/light-Tailed infinitely wide neural networks

Cited 0 time in webofscience Cited 0 time in scopus
73 Viewed 0 Downloaded
Title
α-stable convergence of heavy-/light-Tailed infinitely wide neural networks
Author(s)
Jung, Paul; Lee, Hoil; Lee, Jiho; Hongseok Yang
Publication Date
2023-12
Journal
Advances in Applied Probability, v.55, no.4, pp.1415 - 1441
Publisher
Cambridge University Press
Abstract
We consider infinitely wide multi-layer perceptrons (MLPs) which are limits of standard deep feed-forward neural networks. We assume that, for each layer, the weights of an MLP are initialized with independent and identically distributed (i.i.d.) samples from either a light-Tailed (finite-variance) or a heavy-Tailed distribution in the domain of attraction of a symmetric -stable distribution, where may depend on the layer. For the bias terms of the layer, we assume i.i.d. initializations with a symmetric -stable distribution having the same parameter as that layer. Non-stable heavy-Tailed weight distributions are important since they have been empirically seen to emerge in trained deep neural nets such as the ResNet and VGG series, and proven to naturally arise via stochastic gradient descent. The introduction of heavy-Tailed weights broadens the class of priors in Bayesian neural networks. In this work we extend a recent result of Favaro, Fortini, and Peluchetti (2020) to show that the vector of pre-Activation values at all nodes of a given hidden layer converges in the limit, under a suitable scaling, to a vector of i.i.d. random variables with symmetric -stable distributions, . © The Author(s), 2023. Published by Cambridge University Press on behalf of Applied Probability Trust.
URI
https://pr.ibs.re.kr/handle/8788114/14629
DOI
10.1017/apr.2023.3
ISSN
0001-8678
Appears in Collections:
Pioneer Research Center for Mathematical and Computational Sciences(수리 및 계산과학 연구단) > 1. Journal Papers (저널논문)
Files in This Item:
There are no files associated with this item.

qrcode

  • facebook

    twitter

  • Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.
해당 아이템을 이메일로 공유하기 원하시면 인증을 거치시기 바랍니다.

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Browse