斉藤翔汰，白川真一：適応的ノイズ分布を導入したDeep Neural Networkのための学習法，計測自動制御学会 システム・情報部門 第56回システム工学部会研究会，pp.26-32 (2017).

Abstract

Training deep neural networks (DNNs) often stagnates due to local minima or plateau. To overcome the stagnation of learning, adding gradient noise is proposed. However, the adequate scheduling of the noise variance depends on a model, data, and a cost function. Therefore, we propose a method that adapts the noise variance for a target problem. The proposed method updates the noise variance toward the natural gradient direction to minimize the expected cost function. We apply the proposed method to the DNN training such as for image classification and natural language processing and verify the effect.