December 6, 2018
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
Neural Information Processing Systems (NeurIPS)
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions. For models with many parameters, the covariance matrix they are based on becomes gigantic, making them inapplicable in their original form.
By: Thomas George, Cesar Laurent, Xavier Bouthillier, Nicolas Ballas, Pascal Vincent
Facebook AI Research