The history of AI, 1763 to 2022

Artificial intelligence grew out of probability and statistics long before it had a name. Bayes and Legendre gave it the maths in the 18th and 19th centuries. Turing, McCulloch and Pitts, and the 1956 Dartmouth workshop gave it a goal. The perceptron, backpropagation and decades of datasets and hardware made neural networks work, until AlexNet in 2012 and the Transformer in 2017 led to the language models in use today.

27 milestones, oldest first. Most come with a flashcard so you can check you remember who did what.

1. 1763

   Bayes' theorem is published, after Bayes

   Richard Price publishes Thomas Bayes' essay on inverse probability two years after Bayes' death. The idea of updating a belief with evidence sits under every probabilistic model that followed.

   • Flashcard: Was Bayes's Theorem published after his death?

2. 1805

   Legendre publishes least squares

   Adrien-Marie Legendre describes the method of least squares for fitting orbits to noisy observations. Linear regression is still fitted the same way.

   • Flashcard: The least squares method was published by a famous french mathematician called.

3. 1812

   Laplace's Théorie analytique des probabilités

   Pierre-Simon Laplace states Bayes' theorem in the general form used today, half a century after Bayes' essay.

   • Flashcard: In 1812 Laplace published "Théorie Analytique des Probabilités" in which he defines a famous theorem. Which theorem was it?

4. 1913

   Markov counts the letters of a poem

   Andrey Markov analyses the sequence of vowels and consonants in Pushkin's Eugene Onegin, the first real application of what became Markov chains.

   • Flashcard: Andrey Markov first describes techniques he used to analyse a poem. Later this techniques become known as Markov chains. True or False.

5. 1943

   The first model of an artificial neuron

   Warren McCulloch and Walter Pitts describe a neuron as a threshold unit that fires when its weighted inputs pass a limit, the building block of every neural network since.

6. 1950

   Turing proposes the imitation game

   Alan Turing's paper "Computing Machinery and Intelligence" replaces the question "can machines think?" with a test judged over text: the Turing Test.

   • Flashcard: When was the Turing Test created?

7. 1951

   SNARC, the first neural network machine

   Marvin Minsky and Dean Edmonds build SNARC, a machine of about 40 artificial neurons that learned to find its way through a maze.

   • Flashcard: In 1951 Marvin Minsky and Dean Edmonds conceived what is known as the firts neural network machine. What is it's name?

8. 1956

   The Dartmouth workshop names the field

   John McCarthy, Marvin Minsky, Claude Shannon and Nathaniel Rochester run a summer workshop at Dartmouth College whose proposal coined the term "artificial intelligence".

9. 1958

   Rosenblatt's perceptron

   Frank Rosenblatt publishes the perceptron, a single-layer network that learns its weights from examples. The press coverage promised far more than one layer could deliver.

   • Flashcard: Who invented the perceptron?

10. 1967

    Nearest neighbours gets its theory

    Thomas Cover and Peter Hart publish the error bounds for the nearest neighbor rule, the classifier that simply looks up the closest known example.

    • Flashcard: When was the nearest neighbor algorithm was created?

11. 1970

    Linnainmaa's reverse-mode differentiation

    Seppo Linnainmaa publishes the general method for reverse-mode automatic differentiation, the algorithm behind backpropagation. Rumelhart, Hinton and Williams popularised it for neural networks in 1986.

    • Flashcard: In 1970 Seppo Linnainmaa published the general method for automatic differentiation. This corresponds to the modern version of:

12. 1989

    Q-learning and convolutional nets

    Christopher Watkins introduces Q-learning in his PhD thesis "Learning from Delayed Rewards". The same year Yann LeCun trains a convolutional network with backpropagation to read handwritten zip codes.

    • Flashcard: In 1989 Christopher Watkins published his PhD thesis “Learning from Delayed Rewards” where he introduced the concept of:
    • Flashcard: Who is known of the founding father of convolutional nets?

13. 1993

    R is announced

    Statisticians Ross Ihaka and Robert Gentleman announce R, a free language for statistical computing that became a default tool of data analysis.

    • Flashcard: The R language was designed by statisticians?
    • Flashcard: Who are credited by the creation of R?

14. 1995

    Random forests and support vector machines

    Tin Kam Ho publishes random decision forests, and Corinna Cortes and Vladimir Vapnik publish support-vector networks. Both remain strong baselines on tabular data.

    • Flashcard: In which decade does Tin Kam Ho published a paper describing random decision forests?
    • Flashcard: In 1995 Corinna Cortes and Vladimir Vapnik publish their work explaining a new supervised learning algorithm called:

15. 1997

    Deep Blue beats Kasparov, and the LSTM

    IBM's Deep Blue defeats world chess champion Garry Kasparov in a six-game match. Sepp Hochreiter and Jürgen Schmidhuber publish the LSTM, the recurrent architecture that dominated sequence modelling for two decades.

    • Flashcard: What was the name of the first computer that beat against the the world champion at chess?
    • Flashcard: Who invented the LSTM architecture in deep learning?

16. 1998

    MNIST

    Yann LeCun, Corinna Cortes and Christopher Burges release MNIST, 70,000 images of handwritten digits that became the field's first benchmark.

    • Flashcard: LeCun, Cortes, and Burges compiled a very famous dataset containing 60000 images of handwritten digits. What is the name of this dataset?

17. 2002

    Torch is released

    The Torch machine learning library is released. Its design later carried over to PyTorch.

    • Flashcard: When was Torch released?

18. 2006

    Deep learning gets its name back

    Geoffrey Hinton's work on deep belief networks shows that deep networks can be trained layer by layer, and puts the term "deep learning" at the centre of the field.

    • Flashcard: How coined the term deep learning in 2006?
    • Flashcard: This person’s notable students include Ruslan Salakhutdinov, Ilya Sutskever, and Yann LeCun?

19. 2007

    Theano

    The Montreal Institute for Learning Algorithms releases Theano, one of the first libraries to compile and differentiate numerical expressions on the GPU.

    • Flashcard: What defunct deep learning library was published in 2007 by Montreal Institute for Learning Algorithms?

20. 2009

    ImageNet

    Fei-Fei Li's team presents ImageNet, millions of labelled images across thousands of categories, built on the bet that models were starved of data rather than ideas.

    • Flashcard: Who envisioned and created the ImageNet dataset?

21. 2012

    AlexNet wins ImageNet

    Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton win the ImageNet challenge by a wide margin with a GPU-trained convolutional network written in CUDA and C++. This is the moment deep learning went mainstream.

    • Flashcard: What language was the original AlexNet (ILSVRC 2012) written in?

22. 2014

    Generative adversarial networks

    Ian Goodfellow and colleagues introduce GANs, two networks trained against each other until one can generate convincing samples.

    • Flashcard: Who is knows as the creator of the GANs architecture?

23. 2015

    Keras and TensorFlow

    François Chollet releases Keras in March, and Google open-sources TensorFlow in November. Deep learning becomes something most engineers can try.

    • Flashcard: How is the creator of Keras?
    • Flashcard: When was TensorFlow launched?

24. 2016

    AlphaGo beats Lee Sedol, and PyTorch

    DeepMind's AlphaGo, running on Google's TPUs, beats Lee Sedol 4-1 at Go. At NIPS, Yann LeCun describes intelligence as a cake. Facebook releases PyTorch.

    • Flashcard: What is the name of the computer that beat a professional player at Go?
    • Flashcard: What was the final score of the 2016 match between Google DeepMind’s AlphaGo and Go grandmaster Lee Sedol?
    • Flashcard: In 2016 Google’s AlphaGo won four out of five games of Go against Lee Sedol, a professional Go player from South Korea (9 dan rank). What hardware powered the AlphaGo software to victory?
    • Flashcard: What object did Yann Lecunn use in NIPS 2016 to describe the ’layers’ of intelligence?
    • Flashcard: PyTorch was primarily developed by which company?

25. 2017

    Attention Is All You Need

    Vaswani et al. publish the Transformer, a model built entirely on attention with 512-dimensional embeddings. Every large language model since descends from it.

    • Flashcard: What was the dimension of all the embedding and sublayers of the original Transformer (Attention is All You Need 2017)?

26. 2018

    The Turing Award for deep learning

    Yoshua Bengio, Geoffrey Hinton and Yann LeCun receive the 2018 Turing Award for the conceptual and engineering breakthroughs that made deep neural networks practical.

    • Flashcard: Which computer scientists received the Turing Award for their contributions in deep learning?
    • Flashcard: Who are known as the "Good Parents of AI"?

27. 2022

    ChatGPT

    OpenAI releases ChatGPT in November, a chat interface on a Transformer language model tuned with human feedback, and large language models reach the general public.