A neural network trained to reproduce its input: an encoder compresses it into a small code and a decoder rebuilds it. The input is its own target, so it needs no labels and counts as unsupervised, or self-supervised, learning.
Proposed by @svpino
Card 1 of 264. Answer: A neural network trained to reproduce its input: an encoder compresses it into a small code and a decoder rebuilds it. The input is its own target, so it needs no labels and counts as unsupervised, or self-supervised, learning.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
An autoencoder is a neural network trained to reproduce its input at its output. An encoder f compresses x into a smaller code z, a decoder g rebuilds an approximation x̂ = g(f(x)), and training minimizes a reconstruction loss such as the mean squared error between x and x̂. Each input is its own target, so no labels are needed.
Example: a 28 × 28 MNIST digit has 784 pixel values. An encoder might map them through layers of 256 and 64 units to a 32-number code, and a mirror-image decoder maps it back. The code is 24.5 times smaller than the image, so the network has to learn which features summarize a digit instead of copying pixels. Without a bottleneck or another constraint, it could just learn the identity function; denoising autoencoders instead corrupt the input and keep the clean original as the target.
Are autoencoders supervised? No: they are unsupervised, or more precisely self-supervised. Supervised learning needs input-label pairs (x, y) and minimizes L(f(x), y); an autoencoder minimizes L(g(f(x)), x), with the target taken from the data itself. That made autoencoders a pretraining tool: Hinton and Salakhutdinov's 2006 Science paper trained deep autoencoders on unlabeled data, and their codes beat linear methods such as PCA at compressing images and documents. A classifier can then be trained on the codes with few labels, which is semi-supervised learning.
Uses include dimensionality reduction and anomaly detection, where high reconstruction error flags unusual inputs. A common follow-up: a linear autoencoder trained with squared error learns the same subspace as the top principal components, as Baldi and Hornik showed in 1989, so non-linear activations are what let it do more. Variational autoencoders learn a distribution over codes so that new samples can be generated, and masked autoencoders, which hide 75% of an image's patches and rebuild them, are a modern self-supervised pretraining method.
