Deep Learning interview questions

Neural networks from backpropagation to attention, and the tricks that make them trainable.

56 questions in this category.

These cards group around what you get asked to explain at a whiteboard: activation functions and why a network is a linear model without them, gradient descent and the learning rate that makes or breaks it, dropout and batch normalization, and the architectures that stack the pieces into something useful — CNNs, LSTMs and GRUs, GANs, autoencoders, Transformers.

Type of gradient descent where we use only a single training example to calculate the gradient and update the parameters.

Stochastic Gradient Descent.

Type of gradient descent where we use the whole dataset to perform a single update step.

Batch Gradient Descent.

Type of gradient descent where we use a fixed number of training examples, fewer than the whole dataset, to calculate the gradient.

Mini-batch Gradient Descent.

Name three different deep learning frameworks.

PyTorch, TensorFlow and JAX. Keras is a popular high-level API that runs on top of them; older frameworks include Caffe and Theano.

What is AdaGrad?

A sophisticated gradient descent algorithm that rescales the gradients of each parameter, effectively giving each parameter an independent learning rate.

What are the two actors in a convolution operation?

Convolutional filter and a slice of an input matrix.

How does dropout regularization work in deep learning?

During training, each unit in a layer is set to zero independently with probability p on every forward pass, so each gradient step trains a different random sub-network. At inference no units are dropped.

What is an epoch?

A full training pass over the entire dataset such that each example has been seen once.

What is a batch?

The set of training examples processed together in one forward and one backward pass, producing one weight update. The number of examples in it is the batch size.

Which are the two components of a Generative Adversarial Network?

A generator that turns noise into samples and a discriminator that judges them real or fake, trained against each other in a minimax game. The tension is the point and also the difficulty: unstable training and mode collapse, where the generator serves one convincing output.

What is the function of the generator in a GAN?

It creates new examples.

What is the function of the discriminator in a GAN?

Determine whether examples are real or fake.

What does gradient descent try to minimize?

A loss function.

Which activation function follows these rules: If input is negative or zero, output is 0 and if input is positive, output is equal to input.

ReLU, the rectified linear unit, max(0, x). It is cheap and its gradient is exactly 1 on the positive side, so it does not saturate as sigmoid and tanh do, which is much of why deep stacks became trainable. Its failure mode is dead units stuck at zero.

Explain what a softmax function does.

Provides probabilities for each possible class in a multi-class classification model. The probabilities add up to exactly 1.0

What is learning rate?

Is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.

What does LSTM mean?

Long Short-Term Memory.

What type of layer typically has a cell, an input gate, an output gate and a forget gate?

An LSTM. The cell state carries information along a mostly additive path while the gates learn what to write, forget and read, which is how it survives the vanishing gradients that stall plain RNNs on long sequences. Attention has since displaced it for most sequence work.

Sigmoid squishes values to be between?

0 and 1, which is what makes it the right output layer for a binary probability. Inside a deep network it is a poor choice: gradients vanish once inputs saturate at either tail, and its output is not zero-centred, so ReLU-family activations replaced it in hidden layers.

What does GRU mean?

Gated Recurrent Unit.

Tanh function squishes values to be between?

-1 and 1

Type of layer that typically has a reset gate and an update gate?

A GRU. It folds the LSTM's cell and hidden state into one and drops a gate, so it has fewer parameters and trains faster, usually matching LSTM accuracy on short and medium sequences.

What if we set all the weights of a neural network to 0?

The network can't learn useful features: every unit in a layer computes the same output and gets the same gradient, so the symmetry never breaks. With tanh or ReLU the weights don't move at all; with sigmoid they move in lockstep. Use small random weights instead.

What happens when the learning rate is too large?

Can accelerate the training. However, it is possible that we “shoot” too far and miss the minimum of the function that we want to optimize.

What happens when the learning rate is too small?

Training is slow: tiny steps need many more iterations, and on a fixed budget the model may stop before converging. With noisy mini-batch gradients it does settle closer to a minimum, but it can't step past a poor nearby minimum or a plateau.

Why do we need activation functions?

Without a nonlinear activation, stacked layers collapse into one linear map: W₂(W₁x + b₁) + b₂ is just another linear function of x. Activation functions add the nonlinearity that lets a neural network learn complex functions, such as XOR, that no linear model can represent.

What are the problems with sigmoid as an activation function?

The output of the sigmoid function for large positive or negative numbers is almost zero. From this comes the problem of vanishing gradient.

What is ReLU? How is it better than sigmoid?

ReLU, max(0, x), zeroes negative inputs and passes positive ones through. Unlike sigmoid it doesn't saturate for positive inputs, where its gradient is 1, so it greatly reduces vanishing gradients in deep networks, and it is cheaper to compute.

What’s pooling in CNN? Why do we need it?

Pooling downsamples each feature map by summarizing small windows, usually with the max or the average. It reduces spatial size and computation, lets deeper layers cover a larger part of the input, and makes features somewhat robust to small shifts. It has no learned weights.

Why did the embedding layers and all sub-layers of the original Transformer (Attention Is All You Need, 2017) need the same output dimension?

Because of the residual connections.

One hot encoding is an excellent solution to transform categorical features with high cardinality.

False

The F1 Score is a metric defined as the harmonic mean of precision, recall, and accuracy.

False

As the number of hidden layers increases in a neural network, its capacity also increases.

True

Initializing a neural network’s weights with zero is our best bet to allow the network to converge.

False: with zero weights every unit in a layer computes the same thing and receives the same gradient, so the symmetry never breaks and the layer behaves like a single neuron. Use small random values scaled to the layer's size: He (variance 2/fan_in) for ReLU, Xavier (variance 2/(fan_in + fan_out)) for tanh. Biases can start at zero.

As the dropout ratio used in a neural network increases, the network’s capacity also increases.

False

Stochastic Gradient Descent is the proper technique to use when the full dataset doesn’t fit in memory.

True

Batch Normalization is an efficient backpropagation technique that allows neural networks to learn.

False — it is a normalization layer, not a backpropagation technique. It standardises each activation across the mini-batch, then rescales with learned parameters, which smooths optimisation and allows higher learning rates. Batch statistics also add noise, so it regularises mildly.

Gradient Descent is an algorithm used to minimize overfitting in neural networks.

False

Having high bias means the model is too simple and can’t capture many features during the training phase. This is also known as overfitting.

False

Precision is a performance metric that defines a classification model's ability to identify only relevant samples.

True

Recall is a performance metric that measures a classification model's ability to identify all relevant samples.

True

Softmax is an activation function that always returns the input value if it’s positive and zeroes otherwise.

False

In neural networks, an activation function’s most essential role is to decide whether a unit or neuron should fire.

False. The fire-or-not idea comes from early step-function neurons. An activation function's essential role is to add nonlinearity, without which stacked layers collapse into one linear map.

Using a learning rate that’s too low will cause the training process to be very slow.

True

ReLU is an activation function that, given an input vector, generates an output where the sum of the values in the vector is equal to one.

False

An autoencoder is a neural network that automatically learns by using its inputs as the expected output.

True

Convolutional neural networks are translation invariant.

False as stated: convolution layers are translation equivariant, and pooling makes a CNN only approximately invariant to small shifts.

Autoencoders are one of the most popular supervised learning methods.

False

The type of problem where you need to classify the input into a single class is called multi class classification.

True, when there are three or more classes: each input gets exactly one label. With two classes it's binary classification; when an input can carry several labels at once, it's multi-label classification.

Dropout is an excellent regularization technique that speeds the training process.

False

A neural network with a single layer is capable of approximating any function.

True with conditions: by the universal approximation theorem, one hidden layer with enough units and a non-polynomial activation can approximate any continuous function on a closed, bounded domain. With no hidden layer, a network only draws linear boundaries and can't even fit XOR.

Leaky ReLU is an activation function that lets small negative values pass through when a neuron's input is less than zero.

True

Bagging is the concept of splitting a dataset and randomly placing it into bags for training a model.

False

A categorical feature is a feature that can only take a limited number of possible values.

True

What is intersection over union (IoU)?

A measure of overlap between a predicted region and the ground truth: the area of their intersection divided by the area of their union. It runs from 0 for no overlap to 1 for a perfect match and is the standard score for detection boxes and segmentation masks.

What is focal loss?

A version of cross-entropy that down-weights easy examples: FL = −(1 − p_t)^γ log(p_t), where p_t is the probability given to the true class. With γ > 0, well-classified examples contribute little, so training concentrates on the hard ones; γ = 0 gives back cross-entropy.