AI Agent Engineer

AI Agent Engineer

Optimizers

4 min read

Optimizers

Content and boundary

This page is an original English summary of primary API documentation and original papers. An optimizer defines how parameters update after gradients are given; it cannot repair leakage, incorrect labels, unavailable features, or an inappropriate evaluation split, and no task-independent best default exists.

Establish shared vocabulary first

For an objective , the simplest gradient-descent update is:

where denotes parameters and the learning rate. Different optimizers chiefly change how gradients are smoothed, scaled, or combined with regularization. All still depend on data, loss, batch, numeric precision, and training budget.

Common categories and questions they suit

CategoryCore ideaTradeoff to validate
SGDupdate directly from the current gradientlearning rate, batch, data scale, and convergence speed
SGD + momentum / Nesterovaccumulate historical update direction to reduce local oscillationmomentum, learning-rate schedule, instability/divergence risk
Adamexponentially estimate first- and second-gradient moments with bias correctionlearning rate, betas, numeric-stability term, and generalization
AdamWdecouple weight decay from Adam’s moment estimateswhich parameters decay, learning-rate schedule, and validation benefit
AdaGrad / RMSpropadaptively scale by historical squared gradientssparsity, non-stationary training, and concrete framework implementation

This table is a starting point for diagnosis and experiments, not a selection recipe. For the same task, data volume, pretraining starting point, architecture, mixed precision, batch size, and scheduler can all change results.

Do not conflate Adam, AdamW, and weight_decay

Adam maintains exponential moving estimates of gradients and their squares, commonly abbreviated and , then scales updates with bias-corrected quantities. AdamW applies parameter decay separately from these moment estimates; PyTorch AdamW documentation explicitly describes weight decay as not accumulating into momentum or variance.

Therefore, do not indiscriminately call weight_decay “L2 regularization” across every optimizer and framework. Inspect formulas, parameter groups, and defaults in the current implementation, especially when migrating checkpoints or changing frameworks. Whether to decay bias, normalization-layer, or embedding parameters is also a model/task design choice that needs validation evidence rather than copied parameter groups.

import torch
 
optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=learning_rate,
    weight_decay=weight_decay,
)

These variables are experiment configuration to validate, not recommended constants. A checkpoint should also retain optimizer.state_dict(); it contains optimization state for every parameter, and without it “resume training” is not equivalent to continuing from the interruption.

A reproducible optimization experiment protocol

  1. Fix data version, split unit, baseline, loss, primary metric, and training budget first. Do not choose an optimizer from final-test results.
  2. Change only a few justified variables per round, such as optimizer, learning rate, or scheduler, rather than model, augmentation, and batch all together.
  3. Record Python/framework versions, hardware, randomness settings, precision mode, batch, gradient clipping, loss, learning-rate curve, and validation metrics.
  4. Inspect training loss, validation metrics, error examples, and failure modes together. When training loss falls while validation does not improve, first investigate data, labels, leakage, overfitting, and metric alignment before enlarging the search.
  5. After locking a solution, report results on a sealed test set or new time window. Save model, optimizer, scheduler, random state, and configuration before discussing resumed training or reproducibility.

Common misconceptions

  • Treat learning rate as everything: it matters, but does not replace correct data boundaries or loss/metric alignment.
  • Treat defaults as recommendations: API defaults aim for usability, not an empirical conclusion about your model or business.
  • Look only at final loss: different reduction, batch, and data ranges make loss magnitudes not directly comparable.
  • Save only model weights: inference may resume, but training trajectory, scheduler, and optimizer state can be lost.
  • Judge which optimizer is better on training data: select through validation inside the training region and retain a final test boundary.

References

Review date: 2026-07-22. Interfaces, defaults, and accelerated implementations evolve with framework versions; follow the documentation for the version actually in use before running.

Backlinks