Which neural network activation function can lead to 'dead' neurons because its gradient is zero for all negative inputs?

Show answer

ReLU

The Rectified Linear Unit (ReLU) outputs zero for any negative input, which can lead to neurons that nunca activate or update during training if they fall into that range. Variants like Leaky ReLU or ELU were designed to mitigate this 'dying ReLU' problem by providing a small non-zero gradient for negative values.

Would you remember this in 6 months?

Trivia Forge uses spaced repetition so facts stick. Free, no signup needed.

Turn on Machine Learning practice →

More Machine Learning trivia

Which optimization algorithm combines the benefits of both AdaGrad and RMSProp by using both momentum and adaptive learning rates?
Which regularization technique prevents overfitting by randomly setting a fraction of input neurons to zero during each training step?
Which researcher, who co-authored the 2012 AlexNet paper, later became the Chief Scientist at OpenAI?
Which researcher won the 2018 Turing Award alongside Geoffrey Hinton and Yoshua Bengio for their work on Deep Learning?
Which RNN variant simplifies the LSTM by combining the forget and input gates into a single 'update gate'?
Which specific type of RNN was designed to mitigate the vanishing gradient problem using a 'forget gate'?

See all 98 →