Eroxl's Notes
Definition of a Single Dense Neurone

Warning

These definitions should not be used in practice and are only here to improve understanding of what happens in a dense layer, to implement a dense layer, the equations in here should be used.

Forward Pass

A single dense neurone can be mathematically defined as

  • Definitions
    • is the output of the neurone in the layer
    • is the number of neurones in the previous layer ()
    • is the weight of the connection between the neurone in the previous layer and the current neurone
    • is the bias of the current neurone

This calculation can be easily re-written using a dot product of vectors

  • Definitions
    • is the output of the neurone in the layer
    • is the weight of the connection between the neurones of the previous layer and the current neurone.
    • are the outputs of the previous layer ()
    • is the bias of the current neurone
Backwards Pass
Warning

The dimensions of the matrices in the following equations are probably not compatible for the dot products, and in practice some might need to be transposed

If the layer is not the output layer:

If the layer is the output layer:

  • Definitions
    • is the outputs of the previous layer
    • is the output of the neurone in the layer
    • is the outputs of the next layer
    • is the weight of the connection between the neurones of the previous layer and the current neurone.
    • is the cost function which is trying to be minimised
    • is the number of neurones in layer .