Gradient descent is an iterative optimization algorithm used to find a local minimum in a multivariable function. Gradient descent does this by repeatedly taking small steps in the opposite direction of the gradient of the function.

Gradient descent can be defined as

• Definitions
  • is the current value of the parameters of the algorithm
  • is the learning rate of the algorithm
  • is the gradient of the multivariable function that is being optimized with respect to the parameters

Variations

• Batch Gradient Descent
• Stochastic Gradient Descent
• Mini-Batch Gradient Descent
• Gradient Descent with Momentum
• AdaGrad
• RMSProp
• Adam