Deep learning works very well in practice, but we do not yet understand why. Knowing why networks generalize could explain puzzling behaviours, such as learning from noisy labels, and suggest training methods that memorize less and generalize better.
What we know
Huge networks trained on finite data still work on new examples, contrary to classical statistics.
What blocks us
There is no complete theory explaining why, or predicting when it fails.
Current leads
Scaling laws, infinite-width limits, studies of learning dynamics.
Sub-questions
Why do networks that can memorize randomly labelled data still generalize well on real data?
Among all the solutions that fit the training data, how does gradient descent find one that generalizes well?
Can we design a better measure of how much training examples 'agree' with each other (gradient coherence) that takes the network's structure into account?