Neural Networks and Deep Learning

Why this matters

A neural network sounds mysterious and is not. One neuron does arithmetic a ten-year-old could follow. Everything impressive comes from doing that arithmetic several million times in a particular arrangement, and from one number per connection being adjusted until the answers come out right.

The idea

A network is nodes (neurons) connected in layers, where adjacent layers connect to each other:

Judging dog against cat: the input layer takes the pixels, the hidden layers extract shape and pattern, the output layer says which. How many hidden layers and how many nodes is a design decision, and harder problems need more of both.

Each neuron does three things: multiply each input by a weight, add the results up, and pass the sum through an activation function before handing it on. The weight says how much that input is emphasised, can be negative, and the bigger it is the more that input drives the result. The activation function is what lets the network learn relationships that are not simply proportional.

Training is adjusting all those weights using data. Before training the weights are arbitrary and the predictions are useless. During training, data goes in and the weights shift according to the gap between the prediction and the correct answer. After training, the adjusted weights produce correct answers on data the network never saw. Each neuron stays simple; the capability lives in the combination.

Deep learning stacks many hidden layers. "Deep" just means many. What depth buys is that features build up in stages:

This hierarchy is why deep networks recognise complex patterns accurately. It became practical only once two things arrived together: enough data, and enough computing power, particularly processors suited to AI work such as GPUs and TPUs. Before that, deeper layers could not be trained well enough to help.

The consequence was a leap. Accuracy near or beyond human level arrived in image recognition, speech recognition and natural language processing. The deeper reason is that features used to be designed by hand, one at a time, by people. Deep learning learns them from the data, so it is not limited by what a person thought to specify.

Picture it

flowchart LR
  I["Input layer: pixels"] --> H1["Hidden: lines and colours"]
  H1 --> H2["Hidden: eyes, nose, ears"]
  H2 --> H3["Hidden: whole face"]
  H3 --> O["Output layer: dog or cat"]

Worked example

Why can a network of simple neurons make complex judgments?

Take one neuron deciding nothing more than whether a small patch of image is brighter on the left than the right. It multiplies each pixel by a weight, positive on one side and negative on the other, sums them, and outputs a large number when the pattern matches. That is an edge detector, and it is just weighted addition.

Now give the next layer those edge detectors as inputs. A neuron there can weight several edges at particular angles and fire when they form a corner. Another layer up, corners and curves in the right arrangement become an eye. Another, and eyes plus a snout in the right positions become a face.

Nobody programmed any of that. Training set those weights by repeatedly nudging them toward whatever reduced the error, and the hierarchy fell out because that was the efficient way for the network to get the answers right.

That is the whole trick: simple units, one number per connection, and enough depth for the useful intermediate ideas to form on their own.

Your turn

Try the practice questions for this lesson. Judge statements about layers and training, match each description to its part of the network, and identify what makes deep learning deep.

Take it further

A team wants an app that reads handwritten digits from scanned school registers. Should they write the rules for each digit by hand, or train a deep network on examples?

1. Design the layers. Say what the input layer receives, what the output layer produces, and in one line why hidden layers are needed rather than a direct mapping.
2. Plan the training. Say what the training data must contain, and how you would check the network works on registers it has never seen.
3. Decide. Recommend an approach and give two reasons, referring to why hand-written rules fail here.

Try writing the rule that separates a 1 from a 7 in anyone's handwriting. The difficulty of that is the argument.

Recap

• A network has an input layer, hidden layers that extract features, and an output layer.
• A neuron multiplies inputs by weights, sums them, and applies an activation function.
• Training adjusts the weights from data until new inputs are judged correctly.
• Deep learning stacks many hidden layers so features build from edges to whole objects.
• It needed large data and powerful processors, and it removed the need to design features by hand.