Jason Klusowski’s Post

In forthcoming work at ICML 2024 with Matias Cattaneo and Boris Shigida, we employ backward error analysis to derive ODEs that approximate the optimization trajectories of Adam, enabling us to understand its implicit biases. Unlike gradient descent that implicitly penalizes the 2-norm of the loss gradient, Adam exhibits a distinct behavior in the early stages of training: depending on the hyperparameter settings, it either implicitly penalizes or anti-penalizes the 1-norm of the loss gradient. Operations Research & Financial Engineering Department, Princeton University; Princeton Engineering

I'm looking to hire a postdoc in statistics and machine learning at the Operations Research & Financial Engineering Department, Princeton University, starting summer 2024! Apply here:

I'm thrilled to share our recent paper in collaboration with the brilliant Yan Shuo Tan & exceptionally talented ORFE PhD student, Xin Chen. We formally confirm some of Leo Breiman's empirical findings in his seminal 1996 paper on Stacked Regressions. In particular, we prove that stacking multiple regression models can lead to strictly better predictive accuracy than the top-performing single model selected by metrics like AIC or BIC.