Eroxl's Notes
Definition of a Layer of Dense Neurones

Forward Pass

A whole layer of these dense neurones can be written using a matrix product

  • Definitions
    • are the outputs of the neurones in the layer
    • are the weights of the neurones in the current layer
    • is a row vector containing the biases for each neurone
    • is the activation function performed individually for each entry
    • denotes the matrix transposition (in this case of the weights)
Size of the Matrices in the Forward Pass
  • (the inputs) take the form of a matrix of size where is the number of neurones in the previous layer and is the batch size of the network.
  • (the weights) take the form of a matrix of size where is the number of neurones in the current layer.
  • The output of our current layer with then be a matrix of size .
  • The biases will also take the form of a column vector

For example given the following setup of neurones for layer and layer

Example Dense Network

<svg version="1.1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 312.3537021782604 582.9110761781719" width="312.3537021782604" height="582.9110761781719" filter="invert(93%) hue-rotate(180deg)" class="excalidraw-svg"><symbol id="image-8d4909dc9e6bd888d1753a261a1725e5de75ddd2"><image width="100%" height="100%" href="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiBzdHlsZT0idmVydGljYWwtYWxpZ246IC0wLjAyNWV4OyIgd2lkdGg9IjIzLjU3IiBoZWlnaHQ9IjEwLjI1IiByb2xlPSJpbWciIGZvY3VzYWJsZT0iZmFsc2UiIHZpZXdCb3g9IjAgLTQ0MiA2MDAgNDUzIj48ZGVmcz48cGF0aCBpZD0iTUpYLTQtVEVYLUktMUQ0NUIiIGQ9Ik0yMSAyODdRMjIgMjkzIDI0IDMwM1QzNiAzNDFUNTYgMzg4VDg5IDQyNVQxMzUgNDQyUTE3MSA0NDIgMTk1IDQyNFQyMjUgMzkwVDIzMSAzNjlRMjMxIDM2NyAyMzIgMzY3TDI0MyAzNzhRMzA0IDQ0MiAzODIgNDQyUTQzNiA0NDIgNDY5IDQxNVQ1MDMgMzM2VDQ2NSAxNzlUNDI3IDUyUTQyNyAyNiA0NDQgMjZRNDUwIDI2IDQ1MyAyN1E0ODIgMzIgNTA1IDY1VDU0MCAxNDVRNTQyIDE1MyA1NjAgMTUzUTU4MCAxNTMgNTgwIDE0NVE1ODAgMTQ0IDU3NiAxMzBRNTY4IDEwMSA1NTQgNzNUNTA4IDE3VDQzOSAtMTBRMzkyIC0xMCAzNzEgMTdUMzUwIDczUTM1MCA5MiAzODYgMTkzVDQyMyAzNDVRNDIzIDQwNCAzNzkgNDA0SDM3NFEyODggNDA0IDIyOSAzMDNMMjIyIDI5MUwxODkgMTU3UTE1NiAyNiAxNTEgMTZRMTM4IC0xMSAxMDggLTExUTk1IC0xMSA4NyAtNVQ3NiA3VDc0IDE3UTc0IDMwIDExMiAxODBUMTUyIDM0M1ExNTMgMzQ4IDE1MyAzNjZRMTUzIDQwNSAxMjkgNDA1UTkxIDQwNSA2NiAzMDVRNjAgMjg1IDYwIDI4NFE1OCAyNzggNDEgMjc4SDI3UTIxIDI4NCAyMSAyODdaIi8+PC9kZWZzPjxnIHN0cm9rZT0iY3VycmVudENvbG9yIiBmaWxsPSJjdXJyZW50Q29sb3IiIHN0cm9rZS13aWR0aD0iMCIgdHJhbnNmb3JtPSJzY2FsZSgxLC0xKSI+PGcgZGF0YS1tbWwtbm9kZT0ibWF0aCI+PGcgZGF0YS1tbWwtbm9kZT0ibWkiPjx1c2UgZGF0YS1jPSIxRDQ1QiIgeGxpbms6aHJlZj0iI01KWC00LVRFWC1JLTFENDVCIi8+PC9nPjwvZz48L2c+PC9zdmc+"/></symbol><symbol id="image-0fd09a7ec850d890d8dddcabf1090d0554d546e5"><image width="100%" height="100%" href="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiBzdHlsZT0idmVydGljYWwtYWxpZ246IC0wLjE4NmV4OyIgd2lkdGg9IjUyLjU0IiBoZWlnaHQ9IjE2LjkyIiByb2xlPSJpbWciIGZvY3VzYWJsZT0iZmFsc2UiIHZpZXdCb3g9IjAgLTY2NiAyMzIyLjQgNzQ4Ij48ZGVmcz48cGF0aCBpZD0iTUpYLTMtVEVYLUktMUQ0NUIiIGQ9Ik0yMSAyODdRMjIgMjkzIDI0IDMwM1QzNiAzNDFUNTYgMzg4VDg5IDQyNVQxMzUgNDQyUTE3MSA0NDIgMTk1IDQyNFQyMjUgMzkwVDIzMSAzNjlRMjMxIDM2NyAyMzIgMzY3TDI0MyAzNzhRMzA0IDQ0MiAzODIgNDQyUTQzNiA0NDIgNDY5IDQxNVQ1MDMgMzM2VDQ2NSAxNzlUNDI3IDUyUTQyNyAyNiA0NDQgMjZRNDUwIDI2IDQ1MyAyN1E0ODIgMzIgNTA1IDY1VDU0MCAxNDVRNTQyIDE1MyA1NjAgMTUzUTU4MCAxNTMgNTgwIDE0NVE1ODAgMTQ0IDU3NiAxMzBRNTY4IDEwMSA1NTQgNzNUNTA4IDE3VDQzOSAtMTBRMzkyIC0xMCAzNzEgMTdUMzUwIDczUTM1MCA5MiAzODYgMTkzVDQyMyAzNDVRNDIzIDQwNCAzNzkgNDA0SDM3NFEyODggNDA0IDIyOSAzMDNMMjIyIDI5MUwxODkgMTU3UTE1NiAyNiAxNTEgMTZRMTM4IC0xMSAxMDggLTExUTk1IC0xMSA4NyAtNVQ3NiA3VDc0IDE3UTc0IDMwIDExMiAxODBUMTUyIDM0M1ExNTMgMzQ4IDE1MyAzNjZRMTUzIDQwNSAxMjkgNDA1UTkxIDQwNSA2NiAzMDVRNjAgMjg1IDYwIDI4NFE1OCAyNzggNDEgMjc4SDI3UTIxIDI4NCAyMSAyODdaIi8+PHBhdGggaWQ9Ik1KWC0zLVRFWC1OLTIyMTIiIGQ9Ik04NCAyMzdUODQgMjUwVDk4IDI3MEg2NzlRNjk0IDI2MiA2OTQgMjUwVDY3OSAyMzBIOThRODQgMjM3IDg0IDI1MFoiLz48cGF0aCBpZD0iTUpYLTMtVEVYLU4tMzEiIGQ9Ik0yMTMgNTc4TDIwMCA1NzNRMTg2IDU2OCAxNjAgNTYzVDEwMiA1NTZIODNWNjAySDEwMlExNDkgNjA0IDE4OSA2MTdUMjQ1IDY0MVQyNzMgNjYzUTI3NSA2NjYgMjg1IDY2NlEyOTQgNjY2IDMwMiA2NjBWMzYxTDMwMyA2MVEzMTAgNTQgMzE1IDUyVDMzOSA0OFQ0MDEgNDZINDI3VjBINDE2UTM5NSAzIDI1NyAzUTEyMSAzIDEwMCAwSDg4VjQ2SDExNFExMzYgNDYgMTUyIDQ2VDE3NyA0N1QxOTMgNTBUMjAxIDUyVDIwNyA1N1QyMTMgNjFWNTc4WiIvPjwvZGVmcz48ZyBzdHJva2U9ImN1cnJlbnRDb2xvciIgZmlsbD0iY3VycmVudENvbG9yIiBzdHJva2Utd2lkdGg9IjAiIHRyYW5zZm9ybT0ic2NhbGUoMSwtMSkiPjxnIGRhdGEtbW1sLW5vZGU9Im1hdGgiPjxnIGRhdGEtbW1sLW5vZGU9Im1pIj48dXNlIGRhdGEtYz0iMUQ0NUIiIHhsaW5rOmhyZWY9IiNNSlgtMy1URVgtSS0xRDQ1QiIvPjwvZz48ZyBkYXRhLW1tbC1ub2RlPSJtbyIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoODIyLjIsMCkiPjx1c2UgZGF0YS1jPSIyMjEyIiB4bGluazpocmVmPSIjTUpYLTMtVEVYLU4tMjIxMiIvPjwvZz48ZyBkYXRhLW1tbC1ub2RlPSJtbiIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTgyMi40LDApIj48dXNlIGRhdGEtYz0iMzEiIHhsaW5rOmhyZWY9IiNNSlgtMy1URVgtTi0zMSIvPjwvZz48L2c+PC9nPjwvc3ZnPg=="/></symbol> <!-- svg-source:excalidraw -->

<defs> <style class="style-fonts"> @font-face { font-family: "Virgil"; src: url("https://excalidraw.com/Virgil.woff2"); } @font-face { font-family: "Cascadia"; src: url("https://excalidraw.com/Cascadia.woff2"); } @font-face { font-family: "Assistant"; src: url("https://excalidraw.com/Assistant-Regular.woff2"); } </style>

</defs> <rect x="0" y="0" width="312.3537021782604" height="582.9110761781719" fill="#ffffff"/><g stroke-linecap="round" transform="translate(10.575654836801391 89.07299481830864) rotate(0 41.803062438964844 41.803062438964844)"><path d="M47.8 0.8 C54.99 0.99, 63.72 4.16, 69.21 9.01 C74.71 13.86, 78.62 22.63, 80.78 29.93 C82.93 37.23, 84.14 45.5, 82.13 52.81 C80.12 60.13, 74.25 68.74, 68.71 73.81 C63.18 78.88, 56.04 82.13, 48.92 83.24 C41.8 84.35, 32.99 83.37, 26 80.46 C19.02 77.55, 11.41 72.01, 7.01 65.77 C2.61 59.53, -0.12 50.83, -0.41 43.02 C-0.7 35.2, 1.22 25.46, 5.28 18.89 C9.34 12.31, 15.82 6.26, 23.95 3.56 C32.08 0.85, 48.1 2.45, 54.06 2.64 C60.02 2.84, 60.04 4.08, 59.71 4.71 M37.85 -0.42 C44.77 -1.37, 55.11 2.6, 61.98 6.44 C68.85 10.28, 75.49 16.25, 79.07 22.61 C82.66 28.98, 84.13 37.28, 83.47 44.62 C82.82 51.96, 79.81 60.49, 75.15 66.65 C70.49 72.8, 62.4 79.03, 55.53 81.54 C48.67 84.05, 41.2 83.42, 33.96 81.71 C26.72 80, 17.61 76.58, 12.09 71.27 C6.58 65.95, 2.6 57.02, 0.89 49.83 C-0.83 42.64, -0.72 34.84, 1.78 28.11 C4.28 21.39, 9.61 13.96, 15.91 9.5 C22.21 5.04, 35.89 2.67, 39.57 1.36 C43.26 0.06, 38.05 1.34, 38.04 1.66" stroke="#1e1e1e" stroke-width="2" fill="none"/></g><g stroke-linecap="round" transform="translate(10.740262398137077 224.26713081646818) rotate(0 41.803062438964844 41.803062438964844)"><path d="M46.76 1.01 C53.84 1.17, 62.38 4.51, 68.32 9.24 C74.25 13.97, 79.9 22.2, 82.38 29.4 C84.86 36.6, 85.25 45.19, 83.2 52.44 C81.16 59.7, 75.86 67.97, 70.09 72.95 C64.32 77.93, 55.87 81.2, 48.57 82.33 C41.27 83.47, 33.05 82.48, 26.3 79.79 C19.54 77.09, 12.35 72.36, 8.05 66.17 C3.75 59.99, 0.92 50.34, 0.5 42.66 C0.07 34.98, 1.84 26.6, 5.5 20.08 C9.16 13.56, 15.26 6.65, 22.46 3.54 C29.66 0.43, 44 1.79, 48.68 1.41 C53.35 1.02, 50.4 0.63, 50.5 1.24 M59.95 4.34 C66.59 6.96, 72.17 13.78, 76.31 20.22 C80.46 26.66, 84.95 35.47, 84.84 42.97 C84.72 50.47, 80.09 58.83, 75.63 65.23 C71.17 71.63, 64.95 78.5, 58.09 81.34 C51.22 84.18, 41.74 83.52, 34.42 82.25 C27.1 80.98, 19.71 78.89, 14.15 73.72 C8.59 68.55, 3.2 58.92, 1.07 51.25 C-1.06 43.59, -0.92 34.76, 1.38 27.74 C3.68 20.71, 9.06 13.79, 14.85 9.1 C20.65 4.42, 28.54 0.37, 36.17 -0.38 C43.79 -1.12, 56.48 3.9, 60.58 4.66 C64.69 5.42, 60.92 3.61, 60.79 4.17" stroke="#1e1e1e" stroke-width="2" fill="none"/></g><g stroke-linecap="round" transform="translate(10 361.43683418786236) rotate(0 41.803062438964844 41.803062438964844)"><path d="M48.93 0.56 C56.09 1.19, 63.88 6.07, 69.39 11.12 C74.9 16.18, 79.8 23.7, 82 30.89 C84.21 38.09, 84.76 47.17, 82.63 54.32 C80.5 61.46, 75.21 68.97, 69.21 73.74 C63.21 78.51, 54.12 81.87, 46.63 82.94 C39.13 84.01, 30.92 83.27, 24.25 80.16 C17.59 77.05, 10.68 70.73, 6.65 64.27 C2.63 57.82, -0.08 49.04, 0.1 41.45 C0.28 33.87, 3.46 25.17, 7.72 18.75 C11.98 12.33, 17.31 5.41, 25.68 2.93 C34.06 0.45, 51.4 2.86, 57.96 3.88 C64.53 4.9, 65.41 8.38, 65.06 9.05 M29.37 2.84 C36.03 0.43, 45.13 0.91, 52.5 3.08 C59.87 5.24, 68.37 10.32, 73.6 15.82 C78.83 21.32, 82.72 28.85, 83.89 36.06 C85.05 43.27, 83.96 52.1, 80.59 59.07 C77.23 66.05, 69.92 73.75, 63.68 77.9 C57.44 82.06, 50.73 84.26, 43.17 84.01 C35.6 83.76, 24.83 80.3, 18.27 76.39 C11.72 72.48, 6.93 67.09, 3.83 60.55 C0.73 54.01, -1.15 44.99, -0.32 37.16 C0.51 29.34, 3.81 19.31, 8.83 13.59 C13.84 7.87, 26.26 4.67, 29.76 2.83 C33.26 0.99, 29.4 1.98, 29.82 2.53" stroke="#1e1e1e" stroke-width="2" fill="none"/></g><g stroke-linecap="round" transform="translate(11.181161733774047 489.30495130024224) rotate(0 41.803062438964844 41.803062438964844)"><path d="M59.61 2.91 C66.31 5.58, 72.35 13.38, 76.28 19.74 C80.22 26.1, 83.21 33.55, 83.23 41.06 C83.26 48.57, 80.33 58.48, 76.42 64.81 C72.52 71.14, 66.58 76, 59.81 79.06 C53.05 82.11, 43.34 83.95, 35.83 83.12 C28.32 82.29, 20.47 78.94, 14.73 74.06 C8.98 69.19, 3.46 61.32, 1.36 53.9 C-0.74 46.47, -0.11 36.9, 2.13 29.51 C4.38 22.12, 9.23 14.31, 14.82 9.56 C20.42 4.81, 28.28 1.83, 35.69 0.98 C43.1 0.14, 55.06 3.66, 59.27 4.51 C63.48 5.36, 60.97 5.35, 60.96 6.1 M42.4 1.11 C49.56 1.25, 61.16 4.52, 67.33 8.61 C73.5 12.69, 76.96 18.94, 79.43 25.61 C81.9 32.28, 83.25 41.36, 82.14 48.64 C81.02 55.92, 77.6 63.85, 72.72 69.3 C67.84 74.75, 60.16 79.32, 52.83 81.35 C45.5 83.39, 36.28 84.02, 28.72 81.5 C21.17 78.99, 12.44 72.26, 7.51 66.28 C2.58 60.29, -0.24 52.96, -0.88 45.6 C-1.53 38.24, 0.03 28.77, 3.64 22.11 C7.26 15.45, 14.3 9.23, 20.8 5.64 C27.3 2.05, 38.91 1.19, 42.66 0.58 C46.41 -0.03, 42.93 1.59, 43.31 1.98" stroke="#1e1e1e" stroke-width="2" fill="none"/></g><g stroke-linecap="round" transform="translate(218.58296973899508 154.229890089023) rotate(0 41.803062438964844 41.803062438964844)"><path d="M60.38 3.5 C67.25 6.39, 74.48 14.24, 78.28 20.92 C82.08 27.59, 83.55 36.1, 83.21 43.54 C82.86 50.98, 80.42 59.32, 76.22 65.55 C72.02 71.77, 64.87 78.04, 58.01 80.88 C51.14 83.73, 42.26 83.83, 35.02 82.62 C27.77 81.41, 20.17 78.74, 14.53 73.62 C8.89 68.5, 3.31 59.38, 1.16 51.91 C-1 44.45, -0.59 35.8, 1.59 28.81 C3.77 21.82, 8.47 14.74, 14.23 9.98 C19.98 5.21, 27.99 0.76, 36.12 0.22 C44.25 -0.32, 57.91 5.07, 63 6.72 C68.09 8.37, 66.97 9.73, 66.67 10.11 M26.89 4.49 C33.48 1.19, 42.07 0.44, 49.41 1.5 C56.75 2.56, 65.45 6.08, 70.9 10.84 C76.36 15.6, 80.53 22.58, 82.13 30.05 C83.72 37.53, 83.01 48.27, 80.48 55.7 C77.94 63.14, 72.39 70.15, 66.9 74.69 C61.41 79.23, 55.01 82.11, 47.53 82.95 C40.05 83.79, 28.77 82.92, 22.02 79.74 C15.27 76.55, 10.76 70.38, 7.03 63.85 C3.3 57.31, -0.39 48.14, -0.36 40.54 C-0.33 32.94, 3.07 24.24, 7.21 18.26 C11.35 12.27, 21.53 7.16, 24.49 4.62 C27.45 2.07, 24.42 2.75, 24.98 2.99" stroke="#1e1e1e" stroke-width="2" fill="none"/></g><g stroke-linecap="round" transform="translate(218.7475773003307 289.42402608718254) rotate(0 41.803062438964844 41.803062438964844)"><path d="M25.01 4.74 C31.14 1.44, 39.84 0.19, 47.14 1.13 C54.44 2.07, 63.06 5.49, 68.81 10.37 C74.55 15.26, 79.38 23.36, 81.6 30.46 C83.82 37.56, 84.02 45.68, 82.12 52.96 C80.22 60.24, 75.93 69.22, 70.2 74.13 C64.46 79.05, 55.37 81.52, 47.71 82.44 C40.05 83.36, 31.23 82.56, 24.25 79.66 C17.26 76.75, 9.96 71.29, 5.8 65.01 C1.63 58.72, -0.69 49.49, -0.73 41.95 C-0.76 34.42, 0.72 26.34, 5.61 19.79 C10.49 13.24, 23.95 5.83, 28.59 2.66 C33.22 -0.52, 33.3 0.27, 33.43 0.74 M51.25 1.86 C58.28 2.59, 67.04 7.13, 72.26 12.14 C77.48 17.14, 81.3 24.32, 82.57 31.88 C83.83 39.44, 82.72 50.15, 79.86 57.48 C77 64.81, 71.13 71.57, 65.43 75.85 C59.73 80.12, 53.18 82.62, 45.66 83.13 C38.15 83.63, 26.94 82.36, 20.35 78.87 C13.75 75.39, 9.53 68.92, 6.1 62.22 C2.66 55.52, -0.63 46.25, -0.27 38.67 C0.1 31.08, 3.88 22.52, 8.3 16.73 C12.71 10.94, 19.3 6.74, 26.23 3.92 C33.17 1.11, 45.72 0.4, 49.9 -0.15 C54.08 -0.7, 51.19 -0.49, 51.31 0.61" stroke="#1e1e1e" stroke-width="2" fill="none"/></g><g stroke-linecap="round" transform="translate(218.00731490219368 426.5937294585767) rotate(0 41.803062438964844 41.803062438964844)"><path d="M26.96 3.32 C33.31 0.2, 43.27 -0.31, 50.65 1.3 C58.02 2.92, 65.86 7.78, 71.22 13.01 C76.59 18.24, 81.08 25.26, 82.83 32.67 C84.59 40.08, 84.49 50.45, 81.76 57.45 C79.03 64.45, 72.72 70.36, 66.46 74.66 C60.21 78.96, 51.86 82.55, 44.23 83.25 C36.6 83.94, 27.32 82.43, 20.67 78.83 C14.01 75.22, 7.89 68.25, 4.31 61.62 C0.74 54.99, -1.42 46.46, -0.79 39.06 C-0.16 31.66, 2.89 23.66, 8.11 17.24 C13.33 10.82, 26.18 3.29, 30.51 0.56 C34.84 -2.18, 33.77 -0.14, 34.1 0.84 M58.5 2.99 C65.24 4.83, 71.94 10.38, 75.94 16.75 C79.93 23.11, 82.38 33.36, 82.47 41.16 C82.55 48.96, 79.92 57.27, 76.44 63.53 C72.96 69.79, 68.31 75.35, 61.59 78.72 C54.87 82.09, 43.61 84.42, 36.13 83.78 C28.65 83.13, 22.44 79.66, 16.72 74.87 C11.01 70.09, 4.42 62.26, 1.83 55.06 C-0.76 47.85, -0.53 38.72, 1.2 31.65 C2.93 24.58, 6.83 17.98, 12.2 12.64 C17.56 7.29, 25.68 1.37, 33.38 -0.41 C41.08 -2.19, 54.28 0.99, 58.4 1.94 C62.52 2.89, 58.39 4.55, 58.09 5.3" stroke="#1e1e1e" stroke-width="2" fill="none"/></g><g stroke-linecap="round"><g transform="translate(94.57326331938293 137.62565175779116) rotate(0 61.27669129383929 29.930794042732664)"><path d="M-1.06 -0.02 C19.11 9.98, 100.61 49.74, 121.37 59.61 M0.59 -1.07 C21.07 9.15, 103.02 50.81, 123.46 61.01" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(96.4777625898181 262.8323244768552) rotate(0 60.66102486438483 -31.83165341420016)"><path d="M-0.5 0.63 C19.71 -10.11, 100.66 -52.95, 120.9 -63.75 M1.44 -0.08 C21.49 -10.69, 99.97 -51.76, 119.73 -62.25" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(94.16345975485612 391.64457389967583) rotate(0 61.61369465999685 -96.11339387847025)"><path d="M-0.18 0.59 C20.45 -31.58, 102.48 -160.71, 123.19 -192.8 M-1.73 -0.15 C18.79 -32.18, 101.31 -159.85, 122.22 -191.56" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(95.12659786167836 531.8047548258469) rotate(0 60.956777795041546 -165.98698695938077)"><path d="M-0.43 -0.74 C20.04 -56.01, 102.49 -277.14, 122.97 -332.24 M1.55 1.48 C21.94 -53.55, 102.51 -275.04, 122.57 -330.84" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(96.38317257527963 139.3371626655475) rotate(0 59.98126945913964 94.48215451117181)"><path d="M0.32 -0.15 C20.38 31.33, 100.42 157.95, 120.32 189.48 M-0.97 -1.27 C18.97 29.8, 99.59 155.69, 119.56 187.62" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(98.2876718457148 264.43746454159907) rotate(0 58.747935561369545 30.83404744955986)"><path d="M0.36 0.51 C19.85 10.82, 97.83 51.62, 117.26 61.71 M-0.91 -0.27 C18.38 10.27, 96.72 52.88, 116.19 63.27" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(95.97336901075282 393.2497139644197) rotate(0 60.6048162585587 -32.164282686027164)"><path d="M-0.23 0.04 C19.82 -10.86, 100.34 -54.08, 120.67 -64.93 M-1.81 -0.98 C18 -11.78, 98.93 -53.18, 119.44 -63.7" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(96.93650711757506 533.4098948905907) rotate(0 59.17234963593509 -102.77394026827514)"><path d="M-0.54 -0.6 C19.32 -34.98, 98.94 -172.3, 118.9 -206.26 M1.38 1.7 C21.15 -32.54, 98.65 -170.46, 118.24 -205.09" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(95.77465772870548 531.6896824879793) rotate(0 61.276691293839306 -29.93079404273267)"><path d="M1.15 -0.55 C21.53 -10.47, 102.4 -49.52, 122.54 -59.41 M0.29 1.77 C20.49 -8.52, 101.47 -51.23, 121.58 -61.29" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(97.67915699914064 405.8146004106522) rotate(0 60.66102486438484 31.83165341420017)"><path d="M-0.02 0.45 C19.92 11.15, 100.37 52.98, 120.77 63.64 M-1.48 -0.35 C18.15 10.6, 98.92 54.69, 119.53 65.17" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(95.36485416417867 278.4281118947663) rotate(0 61.61369465999687 96.11339387847028)"><path d="M-0.55 -0.02 C20.16 32.07, 103.77 160.87, 124.28 193 M1.36 -1.08 C22.01 30.63, 103.88 159.32, 123.88 191.29" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g stroke-linecap="round"><g transform="translate(96.32799227100091 137.55714802939957) rotate(0 60.95677779504156 165.9869869593808)"><path d="M1.05 0.77 C21.35 56.2, 101.1 277.29, 121.22 332.52 M0.15 0.14 C20.88 55.2, 103.44 275.11, 123.57 330.68" stroke="#1e1e1e" stroke-width="2" fill="none"/></g></g><mask/><g transform="translate(15.481693131837773 10) rotate(0 42.55914799576121 13.439730946029854)"><use href="#image-0fd09a7ec850d890d8dddcabf1090d0554d546e5" width="85" height="27" opacity="1"/></g><g transform="translate(239.43598100263364 19.2944386058831) rotate(0 21.027585558058995 8.65841758273018)"><use href="#image-8d4909dc9e6bd888d1753a261a1725e5de75ddd2" width="42" height="17" opacity="1"/></g></svg>

with a batch size of the inputs would take the form of

The weights would then take the form

When these are multiplied together we get the following output values

Backwards Pass
Warning

The dimensions of the matrices in the following equations are probably not compatible for dot products, and in practice some might need to be transposed

If the layer is not the output layer:

If the layer is the output layer:

  • Definitions
    • is the outputs of the previous layer
    • is the outputs of the current layer
    • is the outputs of the next layer
    • is the weight matrix for the current layer
    • is the cost function which is trying to be minimised
    • is the number of neurones in layer .