Penalties are terms added to the cost function during training to penalize the algorithm for using large magnitude parameters. This method of penalizing high magnitude parameters can help to prevent overfitting and improve feature selection.
Penalty Types