Exponential decay is similar to time based decay except that it uses an exponent to model the decay.

Exponential learning rate decay can be defined by the following equation

• Definitions
  • is the learning rate for the iteration
  • is the current iteration step
  • is the "decay parameter" and controls how quickly the learning rate will change.