| Learning Rate Schedules For Faster Stochastic Gradient Search (1992) | |||||||||||||||||
Abstract | |||||||||||||||||
| . Stochastic gradient descent is a general algorithm that includes LMS, on-line backpropagation, and adaptive k-means clustering as special cases. The standard choices of the learning rate j (both adaptive and fixed functions of time) often perform quite poorly. In contrast, our recently proposed class of "search then converge" (STC) learning rate schedules (Darken and Moody, 1990b, 1991) display the theoretically optimal asymptotic convergence rate and a superior ability to escape from poor local minima However, the user is responsible for setting a key parameter. We propose here a new methodology for creating the first automatically adapting learning rates that achieve the optimal rate of convergence. INTRODUCTION The stochastic gradient descent algorithm is \DeltaW (t) = \Gammajr W f [W (t); X(t)]; (1) where j is the learning rate, t is the "time", and X(t) is the independent random exemplar chosen at time t. The purpose of the algorithm is to find a parameter vector W that minim... | |||||||||||||||||
Publication details | |||||||||||||||||
| |||||||||||||||||