What is the development of neural networks

Tech blog

In principle, deep learning is the neural network technology that has been introduced for decades, enriched with many additional tricks that make very deep neural networks possible.

In this blog I am concerned with why neural networks did not catch on in the 1960s and 1980s.

The brain cell as a model for electrical circuits

The history of neural networks begins in the 1940s. At that time, researchers examined how neurons work in the brain and tried to recreate them with electrical circuits. The first breakthrough came in 1957 with the Mark I Perceptron, a machine that could use a single “neuron” to divide input data into two classes. She learned by minimizing the error from previous attempts. In the years that followed, researchers tried to use artificial intelligence to solve many different problems, such as natural language. The “STUDENT” program was able to understand mathematical tasks in natural language. However, the number of words in this language was very limited and the algorithm consisted of an optimized search in a graph rather than a learning model. It was similar with the robot Shakey, which got its name because of its trembling movements. Research on this resulted in the A * algorithm, which is still frequently used today, which can efficiently calculate the path between a starting point and a destination point. Even that would no longer be called artificial intelligence today; At the time, however, it helped to drive up expectations of artificial intelligence.

This is what the Shakey robot looked like

Twice hibernation and back

The "Artificial Intelligence Winter" occurred in the 1970s. Further research and funding on artificial intelligence was halted after the exaggerated expectations could not be met. As early as the 1950s, researchers believed that human-like artificial intelligence could be achieved in a few decades. It was thought that if you can teach a machine to play chess, then you can just teach it to do more general tasks.

A TV show with interviews with some of the artificial intelligence pioneers
You don't see a video here? Then watch it directly on YouTube.

In the 1980s, neural networks received more attention with the development of the backpropagation algorithm: this method is used to train multi-layered neural networks by carrying the network's errors back through the network. In the 90s, Yann LeCun, one of the deep learning fathers, developed the first convolutional neural networks to recognize handwritten numbers. This new form of neural networks are particularly suitable for recognizing things in images. Its networks read hundreds of thousands of checks annually. However, there was another forced hibernation for neural networks: they were difficult to train for major problems; other machine learning methods, such as support vector machines, showed good results and took the lead on the artificial intelligence stage.

Until 2012, neural networks were neglected, then there was another breakthrough. Jeffrey Hinton developed a model that shows the error rate in the Large Scale Image Recognition Challenge almost halved. This was made possible by several fundamental innovations in deep learning: Among other things, algorithms were developed to train the networks with graphics cards. Graphics cards are particularly fast when calculating parallel matrix multiplications. Since the training of neural networks mainly consists of matrix multiplications, the training time could be reduced by a factor of 1000; Training times suddenly reduced to an acceptable time.

Tensorflow and Co: graphics cards as calculating machines

Today, frameworks such as Tensorflow, Theano or Pytorch help to execute code particularly easily on graphics cards. Simply means: A developer does not have to write any specialized code that runs on the graphics card; the frameworks translate this for the developer and execute it either on the graphics card or on normal CPUs. The frameworks also transfer data between the graphics card and CPU. They also offer an automatic calculation of the derivative of functions. This is important for the backpropagation algorithm and relieves the developers and researchers of a very complex task in the development of deep learning models.

Excessive expectations?

Deep learning is currently at the peak of exaggerated expectations in the Gartner Hype Cycle. The hypecycle summarizes that new technologies usually cannot meet the expectations of the first hype; they must first cross the valley of disappointment to ultimately find application in industry. Deep learning is therefore - at least according to this model - on the verge of disappointment.

However, the underlying neural networks have already left the valley of disappointment behind them at least twice: in the 1960s, funding for research on them and other models of artificial intelligence was stopped after the extremely high expectations could not be met. After their resurrection in the 80s, neural networks could not prevail against other methods and took a back seat. But some researchers did not give up hope for the algorithm based on the model from nature. If our brains lead to intelligence, then it must be possible to recreate this structure with machines, so the thought. The persistence of the researchers has paid off: Even if human-like intelligence has not yet been achieved by far, neural networks are showing incredible progress in many areas. Be it recognizing things in pictures, understanding or outputting natural language, or translating languages ​​- all of this is possible today. It was recently demonstrated that computers can be trained with deep learning algorithms to beat human opponents in complex team-based strategy games (article on openai.com. And deepmind.com). It is also no longer just the tech giants who use deep learning technology to solve complex problems. The technology has already been adopted in many other companies.

I'm curious to see whether deep learning technology will have to go through the valley of disappointment again, or whether the neural networks' hibernation periods are already sufficient.

Tweet Share Share on LinkedIn