The rectified linear function is an activation function that holds on to the positive values of the input and sets the negative values to zero.

Advantages

• It's is a more informative than the step function.

Potential Disadvantages

• Not differentiable at zero (Usually the derivative at zero is set to either zero or one).
• It's an unbounded function which can lead to exploding gradients.
• The dying RELU problem

The rectified linear function can be defined using the following piecewise function

Derivative

This derivative is technically undefined at so usually in practice it is set to either or at .

Variants

• Leaky ReLU Function
• Parametric ReLU Function