AdamOptimizer.h File Reference

Vectorized AdamW update, operating per-element over an array of weights (or biases) – NOT the single-scalar-averaging design from the earlier draft, which would have collapsed many distinct per-weight gradients into one shared update and thrown away exactly the per-parameter information Adam depends on. More...

#include <cstddef>
#include <cmath>
#include <immintrin.h>
#include <sleef.h>
#include <cstring>

Go to the source code of this file.

Namespaces
namespace	Deep
	This file exists to provide both Tensor.h and IComputeBackend.h with device types without introducing circular dependencies.

Macros
#define	RESTRICT   __restrict__
	Portability macro for the C++ restrict pointer qualifier (MSVC spells it differently from everyone else).

Enumerations
enum class	Deep::OptimizerType { Deep::SGD , Deep::ADAM , Deep::ADAMW }
	Selects which weight-update rule a layer applies during UpdateWeights(). More...

Functions
OptimizerType	Deep::toOpType (const char *s)
	Parses an OptimizerType from its name.
void	Deep::AdamWUpdate (float *RESTRICT w, const float *RESTRICT grad, float *RESTRICT m, float *RESTRICT v, size_t n, int t, float lr, float lambda, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f) noexcept
	Vectorized AdamW update over n elements. w, grad, m, v are all arrays of length n – ONE m/v pair PER WEIGHT, not one shared scalar. t is the layer's own timestep counter (increment ONCE per UpdateWeights() call, not per element – shared across the whole array, which is why beta1^t/beta2^t are computed once here, not per element).
void	Deep::AdamUpdate (float *RESTRICT w, const float *RESTRICT grad, float *RESTRICT m, float *RESTRICT v, size_t n, int t, float lr, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f) noexcept
	Vectorized PLAIN Adam update over n elements – NO weight decay term at all, unlike AdamWUpdate above. This is deliberate, not a missing feature: "Adam with L2 regularization" (folding decay into the gradient before the moment estimates see it) is the well-known WORSE approach AdamW was specifically invented to fix – large-gradient weights get decayed less than they should, since the adaptive step size shrinks the decay term along with everything else. If you want regularization alongside plain Adam, apply this codebase's existing decay convention separately (e.g. cblas_sscal(n, 1.0f-lmbda, w, 1) before calling this) – fully composable, and avoids re-introducing the exact problem AdamW exists to solve.

Detailed Description

Vectorized AdamW update, operating per-element over an array of weights (or biases) – NOT the single-scalar-averaging design from the earlier draft, which would have collapsed many distinct per-weight gradients into one shared update and thrown away exactly the per-parameter information Adam depends on.

IMPORTANT – this requires the RAW, unscaled gradient, not the already-lr-scaled delta the existing UpdateWeights() GEMM calls produce. The GEMM's alpha parameter needs to change from lr/batchSize to 1.0, writing into a scratch gradient buffer, with THIS function handling all scaling (lr, momentum, bias correction) afterward. Adam's moment estimates (m, v) are only meaningful if they're tracking the raw gradient's own magnitude consistently across timesteps – baking lr scaling into the gradient before Adam sees it would corrupt that.

lr is a genuine runtime parameter, not a compile-time constant – this session has repeatedly found lr needs to be tunable per-layer and per schedule (base rate, decay, and a found-empirically-necessary correction factor for at least one training regime); hardcoding it would silently break all of that.

Note

Each SIMD width (AVX-512/AVX2+AVX/SSE) is handled as a separate preprocessor branch with its own scalar-tail fallback, so exactly one of the vectorized paths compiles in per target plus the shared scalar loop for any remainder elements not divisible by the vector width.

Version

1.0

Date

2026-06-30

Author

Jack Rose

Macro Definition Documentation

RESTRICT

#define RESTRICT   __restrict__

Portability macro for the C++ restrict pointer qualifier (MSVC spells it differently from everyone else).