Learning Dynamics

Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension

SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training

Gradient Multi-Normalization for Stateless and Scalable LLM Training