Which optimization algorithm combines the benefits of both AdaGrad and RMSProp by using both momentum and adaptive learning rates?

Show answer

Adam

Adam, or Adaptive Moment Estimation, computes adaptive learning rates for each parameter by storing an exponentially decaying average of past gradients and squared gradients. This makes it robust for non-stationary objectives and noisy gradients.

Would you remember this in 6 months?

Trivia Forge uses spaced repetition so facts stick. Free, no signup needed.

Turn on Machine Learning practice →

More Machine Learning trivia

Which regularization technique prevents overfitting by randomly setting a fraction of input neurons to zero during each training step?
Which researcher, who co-authored the 2012 AlexNet paper, later became the Chief Scientist at OpenAI?
Which researcher won the 2018 Turing Award alongside Geoffrey Hinton and Yoshua Bengio for their work on Deep Learning?
Which RNN variant simplifies the LSTM by combining the forget and input gates into a single 'update gate'?
Which specific type of RNN was designed to mitigate the vanishing gradient problem using a 'forget gate'?
Which technique improves neural network training by normalizing layer inputs to have a mean of zero and a variance of one for each mini-batch?

See all 98 →