This file exists to provide both Tensor.h and IComputeBackend.h with device types without introducing circular dependencies. More...

Namespaces
namespace	detail

Classes
class	CPUBackend
	CPU implementation of IComputeBackend: SIMD (via Activations.h) for elementwise ops, BLAS for GEMM. Graph capture/replay are no-ops (there's no async device queue to record). More...
class	CUDABackend
	CUDA implementation of IComputeBackend: device kernels for elementwise ops, cuBLAS for GEMM, CUTLASS for the fused forward path, CUDA graphs for capture/replay. See src/backend/cuda/ for the implementation, split by concern to mirror this interface's own section grouping. More...
class	IComputeBackend
	Device-agnostic compute backend interface: every numerical primitive a layer/network needs, implemented once each by CPUBackend (SIMD/BLAS) and CUDABackend (kernels/cuBLAS/CUTLASS), so layer code never branches on device directly. More...
class	Tensor
	A device-aware, RAII-managed, move-only buffer – see the file-level note above for the full ownership/device contract. More...
class	ConvPCLayer
	Convolutional Predictive Coding layer (im2col-based). See the file-level warning above before enabling an unclamped middle layer. More...
class	DirectKPPCLayer
	Predictive Coding layer using Direct Kolen-Pollack feedback alignment (DFA): a per-layer feedback matrix Psi, trained to gradually correlate with W's transpose chain, replaces symmetric weight transport for the feedback term. See DirectKPPCNetwork.h's class docs and the paper's Appendix A.1. More...
class	DiscriminativePCLayer
class	FullPCLayer
	PC layer combining muPC scaling, optional residual connections, and DKP direct feedback – every extra OFF by default, so plain defaults reproduce DirectKPPCLayer exactly. See the file-level note above for the full design rationale. More...
class	GaussSeidelPCLayer
	PC layer with Gauss-Seidel (sequential-sweep) settling dynamics, in place of the usual simultaneous update. Not yet gradient-checked – see the file-level warning above. More...
struct	TensorDescriptor
	A named, shaped view into a layer's weight/bias data, for ModelIO's save/load (see Layer::GetStateDict()). More...
class	Layer
	A deepity layer virtual class. More...
class	RBLayer
	Restricted Boltzmann-style Predictive Coding (PC) Layer. More...
class	SimpleConvPCLayer
	Convolutional PC layer, precision-free, AdamW-capable, routed through IComputeBackend for GPU portability – see the file-level warning above for the GPU path's verification status. More...
class	SimplePCLayer
	Predictive Coding layer without precision weighting – the simpler, unweighted-energy counterpart to DiscriminativePCLayer. More...
class	ConvPCNetwork
	Convolutional Predictive Coding Network built from ConvPCLayer. More...
class	DirectKPPCNetwork
	Predictive Coding network built from DirectKPPCLayer, using Direct Kolen-Pollack feedback alignment (DFA) instead of symmetric weight transport. See the file-level warning above on its current verification status. More...
class	DiscriminativePCNetwork
	An abstracted class for an array of DiscriminativePCLayer. More...
class	FullPCNetwork
	Predictive Coding network combining muPC scaling, optional residual connections, and DKP direct feedback – see the file-level note above for how its defaults reproduce DirectKPPCNetwork exactly. More...
class	GaussSeidelPCNetwork
	Predictive Coding network built from GaussSeidelPCLayer, sweeping layers sequentially rather than updating them simultaneously. See the file-level warning above on its current verification status. More...
class	SimpleConvPCNetwork
	Convolutional Predictive Coding Network built from SimpleConvPCLayer – precision-free, AdamW-capable, device-aware. More...
class	SimplePCNetwork
	An abstracted class for an array of SimplePCLayer. More...
class	DeviceMemoryArena
	A 64-byte-aligned bump-pointer allocator over a single contiguous block of device memory, obtained via a supplied IComputeBackend (expected to be a CUDABackend in practice, though this class itself doesn't hard-require that specific type). More...
class	MemoryArena
	A 64-byte-aligned bump-pointer allocator over a single contiguous block of host memory. Every chunk handed out via AllocateFloats() is itself 64-byte aligned, satisfying this codebase's strict alignment requirement for AVX2/AVX-512 loads. More...
class	ModelIO
	Static save/load helpers for any network type exposing GetLayers() – templated so it works across every PC network variant without each one reimplementing serialization. More...
struct	ProfileAccumulator
	One layer's accumulated time-per-phase, in seconds – see the file-level note for the reconciliation invariants (PhaseSum() vs. the top-level totals vs. external wall-clock). More...
struct	ProfileEntry
	One registered layer's profiling data: its print label paired with its accumulator. More...
class	ScopedTimer
	No-op stand-in for the PCN_PROFILE build of ScopedTimer above, so PCN_TIME() call sites compile identically either way. More...
class	StreamAlignedBatcher
	Builds batches with a fixed, grouped number of examples per class – see the file-level note for why this exists and what it deliberately doesn't own. More...

Typedefs
using	ActivationFn = void ()(float , size_t)
	Pointer to an in-place activation function: f(buf, n).
using	DerivativeFn = void ()(float , size_t, bool)
	Pointer to an in-place activation derivative: f(buf, n, activated).
using	DerivativeFn2 = void ()(float RESTRICT, const float *RESTRICT, size_t)
	Pointer to a two-buffer activation derivative: reads src, writes the derivative into dst, leaving src untouched.

Enumerations
enum class	DeviceType { DEVICE_CPU , DEVICE_GPU }
	Which physical device a Tensor/IComputeBackend pair runs on. More...
enum class	ActivationType : uint8_t { RELU , dRELU , GELU , dGELU , SIGMOID , dSIGMOID , eSIGMOID , d_eSIGMOID , TANH , dTANH , LINEAR , dLINEAR , NONE }
	Identifies an activation function or its derivative, for dispatch tables and layer/backend APIs that need to name an activation without depending on the SIMD implementations themselves (see the file-level note above on why this is split out of Activations.h). More...
enum class	OptimizerType { SGD , ADAM , ADAMW }
	Selects which weight-update rule a layer applies during UpdateWeights(). More...

Functions
std::unique_ptr< IComputeBackend >	CreateBackend (DeviceType device)
	Factory function to create the appropriate backend.
static void	ProtectFPU ()
	Flushes denormal floats to zero on both input and output of SSE FP ops, avoiding the severe slowdown x86 FPUs hit computing with denormals – settling can drive values arbitrarily close to zero over many steps.
static void	relu (float *, size_t) noexcept
	See Relu.h.
static void	gelu (float *, size_t) noexcept
	See Gelu.h.
static void	sigmoid (float *, size_t) noexcept
	See Sigmoid.h.
static void	e_sigmoid (float *, size_t) noexcept
	See Sigmoid.h.
static void	tanh (float *, size_t) noexcept
	See Tanh.h.
static void	linear (float *, size_t) noexcept
	See Linear.h.
static void	dRelu (float *, size_t, bool) noexcept
	See Relu.h.
static void	dGelu (float *, size_t, bool) noexcept
	See Gelu.h.
static void	dSigmoid (float *, size_t, bool) noexcept
	See Sigmoid.h.
static void	d_eSigmoid (float *, size_t, bool) noexcept
	See Sigmoid.h.
static void	dTanh (float *, size_t, bool) noexcept
	See Tanh.h.
static void	dLinear (float *, size_t, bool) noexcept
	See Linear.h.
static void	dReluInto (float RESTRICT, const float RESTRICT, size_t) noexcept
	See Relu.h.
static void	dGeluInto (float RESTRICT, const float RESTRICT, size_t) noexcept
	See Gelu.h.
static void	dSigmoidInto (float RESTRICT, const float RESTRICT, size_t) noexcept
	See Sigmoid.h.
static void	d_eSigmoidInto (float RESTRICT, const float RESTRICT, size_t) noexcept
	See Sigmoid.h.
static void	dTanhInto (float RESTRICT, const float RESTRICT, size_t) noexcept
	See Tanh.h.
static void	dLinearInto (float RESTRICT, const float RESTRICT, size_t) noexcept
	See Linear.h.
static ActivationFn	To_Fn (ActivationType type)
	Maps an ActivationType to its concrete function pointer.
static DerivativeFn	To_dFn (ActivationType dType)
	Maps an ActivationType (a derivative variant, e.g. dRELU) to its concrete function pointer.
static DerivativeFn2	To_dFn2 (ActivationType dType)
	Dispatches to the two-buffer (dst, src, n) derivative variant for the given activation type – see DerivativeFn2.
static ActivationType	To_AType (ActivationFn fn)
	Reverse lookup of To_Fn(): maps a function pointer back to its ActivationType, by identity comparison against every known activation function.
static ActivationType	To_AType (DerivativeFn dfn)
	Same as the ActivationFn overload above, for derivative function pointers.
static void	gelu (float *RESTRICT x, const size_t n) noexcept
	In-place GELU activation (tanh approximation, ~5e-4 max absolute error vs. true GELU – see tests/tGelu.cpp).
static void	dGelu (float *RESTRICT x, const size_t n, const bool activated) noexcept
	In-place GELU derivative.
static void	dGeluInto (float RESTRICT dst, const float RESTRICT src, const size_t n) noexcept
	Two-buffer, single-pass GELU derivative: reads src (the pre-activation value), writes the derivative directly into dst.
static void	linear (float *x, size_t n) noexcept
	Identity activation: f(x) = x. A no-op, kept as a real function so it can be used interchangeably with the other activations through ActivationFn.
static void	dLinear (float *x, size_t n, bool activated=false) noexcept
	Derivative of linear(): a constant 1 everywhere.
static void	dLinearInto (float RESTRICT dst, const float RESTRICT src, size_t n) noexcept
	Two-buffer variant of dLinear – src is unused (the derivative of a linear function is a constant), kept for signature consistency with To_dFn2's dispatch table.
static void	relu (float *RESTRICT x, const size_t n) noexcept
	RELU(x) = MAX(0, x) for all x.
static void	dRelu (float *RESTRICT x, const size_t n, bool activated=false) noexcept
	In-place ReLU derivative: 1 where x > 0, else 0.
static void	dReluInto (float RESTRICT dst, const float RESTRICT src, const size_t n) noexcept
	Two-buffer variant of dRelu: reads src, writes the derivative directly into dst.
static void	sigmoid (float *RESTRICT x, const size_t n) noexcept
	Implements the Logistic Sigmoid approximation, i.e. S(x) = 1 / (1 + e^(-x)).
void	e_sigmoid (float *RESTRICT x, const size_t n) noexcept
	Implements the Elliot Sigmoid approximation, i.e. S(x) = (1/2)((x / (1 + \|x\|)) + 1).
static void	dSigmoid (float *RESTRICT x, const size_t n, bool activated=false) noexcept
	In-place logistic sigmoid derivative: sigmoid(x)*(1-sigmoid(x)).
static void	dSigmoidInto (float RESTRICT dst, const float RESTRICT src, const size_t n) noexcept
	Two-buffer, single-pass sigmoid derivative: reads src, writes sigmoid(src)*(1-sigmoid(src)) directly into dst. Computes the sigmoid into a register and derives from that same register immediately – one read of src, one write to dst, no intermediate array pass.
static void	d_eSigmoid (float *RESTRICT x, const size_t n, const bool activated=false) noexcept
	In-place Elliot sigmoid derivative.
static void	d_eSigmoidInto (float RESTRICT dst, const float RESTRICT src, const size_t n) noexcept
	Two-buffer, single-pass Elliot sigmoid derivative: reads src, writes the derivative directly into dst.
static void	tanh (float *RESTRICT x, const size_t n) noexcept
	In-place hyperbolic tangent activation.
static void	dTanh (float *RESTRICT x, const size_t n, const bool activated=false) noexcept
	In-place tanh derivative: 1 - tanh(x)^2.
static void	dTanhInto (float RESTRICT dst, const float RESTRICT src, const size_t n) noexcept
	Two-buffer, single-pass tanh derivative: reads src, writes 1-tanh(src)^2 directly into dst. Unlike dTanh(x, n, false), which needs two full passes internally (activate in place, then derive in place) plus a separate copy beforehand if src can't be mutated, this computes tanh(src[i]) into a register and derives from that same register immediately, with only one read of src and one write to dst – no intermediate array pass at all.
static void	expf_v (float *x, size_t n)
	In-place vectorized exp: x[i] = exp(x[i]) for all i.
void	logf_v (float *x, size_t n)
	In-place vectorized log: x[i] = log(x[i]) for all i.
OptimizerType	toOpType (const char *s)
	Parses an OptimizerType from its name.
void	AdamWUpdate (float RESTRICT w, const float RESTRICT grad, float RESTRICT m, float RESTRICT v, size_t n, int t, float lr, float lambda, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f) noexcept
	Vectorized AdamW update over n elements. w, grad, m, v are all arrays of length n – ONE m/v pair PER WEIGHT, not one shared scalar. t is the layer's own timestep counter (increment ONCE per UpdateWeights() call, not per element – shared across the whole array, which is why beta1^t/beta2^t are computed once here, not per element).
void	AdamUpdate (float RESTRICT w, const float RESTRICT grad, float RESTRICT m, float RESTRICT v, size_t n, int t, float lr, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f) noexcept
	Vectorized PLAIN Adam update over n elements – NO weight decay term at all, unlike AdamWUpdate above. This is deliberate, not a missing feature: "Adam with L2 regularization" (folding decay into the gradient before the moment estimates see it) is the well-known WORSE approach AdamW was specifically invented to fix – large-gradient weights get decayed less than they should, since the adaptive step size shrinks the decay term along with everything else. If you want regularization alongside plain Adam, apply this codebase's existing decay convention separately (e.g. cblas_sscal(n, 1.0f-lmbda, w, 1) before calling this) – fully composable, and avoids re-introducing the exact problem AdamW exists to solve.
int	ConvOutDim (int inDim, int kernel, int stride, int pad) noexcept
	Computes a single convolution output dimension (height or width) from its corresponding input dimension and conv parameters.
void	Im2Col (const float input, int channels, int height, int width, int kH, int kW, int strideH, int strideW, int padH, int padW, float columns) noexcept
	Rearranges a single (channels, height, width) input image into a (channelskHkW, outHoutW) column matrix, where each column holds one flattened receptive-field patch – the standard im2col transform that lets convolution be computed as a single GEMM against a (outChannels, channelskH*kW) weight matrix.
void	Col2Im (const float columns, int channels, int height, int width, int kH, int kW, int strideH, int strideW, int padH, int padW, float outputImage) noexcept
	The adjoint of Im2Col(): scatters a (channelskHkW, outH*outW) column-gradient buffer back into a (channels, height, width) image.
size_t	GetL2CacheBytes ()
	Queries the size of this CPU's L2 cache (per-core, level 2), falling back to 512 KiB on platforms where it can't be queried.
size_t	AutoBatchSize (size_t inSize, size_t outSize)
	Picks a batch size (a power of 2 clamped to [64, 512]) sized so a layer's weights plus one batch's activations fit within the L2 cache, for a size -> outSize transition.
static void	DynamicThread (int batchSize) noexcept
	Sets OpenMP's (and the BLAS library's) thread count for the given batch size, memoized so it's a no-op if the target hasn't changed since the last call. Small batches get 1 thread (avoids parallelization overhead dominating tiny work); larger batches scale up to a measured, capped optimum – see the inline note on why 16 threads, not the full core count, is the ceiling.
static float	hsum256_ps (__m256 x)
	Horizontal sum of all 8 lanes of an AVX2 register.
static float	hsum128_ps (__m128 x)
	Horizontal sum of all 4 lanes of an SSE register.
std::vector< std::pair< const void *, ProfileEntry > > &	ProfileRegistry ()
	The process-wide table of every layer that's had LayerProfile() called on it at least once, in first-call order.
template<typename LayerT>
ProfileAccumulator &	LayerProfile (LayerT *layer)
	Looks up (or creates, on first call) layer's accumulator in the process-wide registry, keyed by its address.
void	PrintAllProfiles (double wallClockSeconds)
	Prints every registered layer's profile, then a grand-total reconciliation against an independently measured wall-clock time.

Variables
constexpr float	MAGIC_GELU_1 = 0.7978845608028654f
	sqrt(2/pi), the leading constant in the tanh approximation of GELU. Also used directly by CUDABackend's device kernels.
constexpr float	MAGIC_GELU_2 = 0.044715f
	The tanh approximation's cubic-term coefficient (0.044715), paired with MAGIC_GELU_1.

Detailed Description

This file exists to provide both Tensor.h and IComputeBackend.h with device types without introducing circular dependencies.

Typedef Documentation

◆ ActivationFn

using Deep::ActivationFn = void (*)(float *, size_t)

Pointer to an in-place activation function: f(buf, n).

◆ DerivativeFn

using Deep::DerivativeFn = void (*)(float *, size_t, bool)

Pointer to an in-place activation derivative: f(buf, n, activated).

◆ DerivativeFn2

using Deep::DerivativeFn2 = void (*)(float *RESTRICT, const float *RESTRICT, size_t)

Pointer to a two-buffer activation derivative: reads src, writes the derivative into dst, leaving src untouched.

Enumeration Type Documentation

◆ ActivationType

enum class Deep::ActivationType : uint8_t

strong

Identifies an activation function or its derivative, for dispatch tables and layer/backend APIs that need to name an activation without depending on the SIMD implementations themselves (see the file-level note above on why this is split out of Activations.h).

Enumerator
RELU
dRELU
GELU
dGELU
SIGMOID
dSIGMOID
eSIGMOID
d_eSIGMOID
TANH
dTANH
LINEAR
dLINEAR
NONE

◆ DeviceType

enum class Deep::DeviceType

strong

Which physical device a Tensor/IComputeBackend pair runs on.

Enumerator
DEVICE_CPU
DEVICE_GPU

◆ OptimizerType

enum class Deep::OptimizerType

strong

Selects which weight-update rule a layer applies during UpdateWeights().

Enumerator
SGD	Plain stochastic gradient descent (this codebase's existing cblas-based update).
ADAM	Adam with no decoupled weight decay; see AdamUpdate().
ADAMW	Adam with decoupled weight decay; see AdamWUpdate().

Function Documentation

◆ AdamUpdate()

void Deep::AdamUpdate	(	float *RESTRICT	w,
		const float *RESTRICT	grad,
		float *RESTRICT	m,
		float *RESTRICT	v,
		size_t	n,
		int	t,
		float	lr,
		float	beta1 = 0.9f,
		float	beta2 = 0.999f,
		float	eps = 1e-8f )

inlinenoexcept

Vectorized PLAIN Adam update over n elements – NO weight decay term at all, unlike AdamWUpdate above. This is deliberate, not a missing feature: "Adam with L2 regularization" (folding decay into the gradient before the moment estimates see it) is the well-known WORSE approach AdamW was specifically invented to fix – large-gradient weights get decayed less than they should, since the adaptive step size shrinks the decay term along with everything else. If you want regularization alongside plain Adam, apply this codebase's existing decay convention separately (e.g. cblas_sscal(n, 1.0f-lmbda, w, 1) before calling this) – fully composable, and avoids re-introducing the exact problem AdamW exists to solve.

Same per-element m/v semantics as AdamWUpdate: one m/v pair PER WEIGHT, t is the layer's own shared timestep counter (increment ONCE per UpdateWeights() call, not per element).

Parameters

w	Weight (or bias) array of length `n`; updated in place.
grad	RAW, unscaled gradient array of length `n` (NOT pre-scaled by lr or batch size – see file-level warning).
m	First-moment (mean) estimate array of length `n`; updated in place. One entry per weight.
v	Second-moment (uncentered variance) estimate array of length `n`; updated in place. One entry per weight.
n	Number of elements in `w` / `grad` / `m` / `v`.
t	This layer's shared timestep counter, incremented once per UpdateWeights() call (not per element).
lr	Learning rate.
beta1	Exponential decay rate for the first-moment estimate. Defaults to 0.9.
beta2	Exponential decay rate for the second-moment estimate. Defaults to 0.999.
eps	Numerical-stability constant added to the denominator. Defaults to 1e-8.

◆ AdamWUpdate()

void Deep::AdamWUpdate	(	float *RESTRICT	w,
		const float *RESTRICT	grad,
		float *RESTRICT	m,
		float *RESTRICT	v,
		size_t	n,
		int	t,
		float	lr,
		float	lambda,
		float	beta1 = 0.9f,
		float	beta2 = 0.999f,
		float	eps = 1e-8f )

inlinenoexcept

Vectorized AdamW update over n elements. w, grad, m, v are all arrays of length n – ONE m/v pair PER WEIGHT, not one shared scalar. t is the layer's own timestep counter (increment ONCE per UpdateWeights() call, not per element – shared across the whole array, which is why beta1^t/beta2^t are computed once here, not per element).

AdamW's decoupled weight decay (w -= lr*lambda*w, applied separately from the gradient-based step) replaces this codebase's existing cblas_sscal(..., 1.0f-lmbda, W, 1) weight-decay call – don't apply both, or decay will be applied twice.

Parameters

w	Weight (or bias) array of length `n`; updated in place.
grad	RAW, unscaled gradient array of length `n` (NOT pre-scaled by lr or batch size – see file-level warning).
m	First-moment (mean) estimate array of length `n`; updated in place. One entry per weight.
v	Second-moment (uncentered variance) estimate array of length `n`; updated in place. One entry per weight.
n	Number of elements in `w` / `grad` / `m` / `v`.
t	This layer's shared timestep counter, incremented once per UpdateWeights() call (not per element).
lr	Learning rate.
lambda	Decoupled weight-decay coefficient (AdamW-style, applied directly to `w`, not folded into the gradient).
beta1	Exponential decay rate for the first-moment estimate. Defaults to 0.9.
beta2	Exponential decay rate for the second-moment estimate. Defaults to 0.999.
eps	Numerical-stability constant added to the denominator. Defaults to 1e-8.

◆ AutoBatchSize()

size_t Deep::AutoBatchSize	(	size_t	inSize,
		size_t	outSize )

inline

Picks a batch size (a power of 2 clamped to [64, 512]) sized so a layer's weights plus one batch's activations fit within the L2 cache, for a size -> outSize transition.

Parameters

inSize	Layer input width.
outSize	Layer output width.

◆ Col2Im()

void Deep::Col2Im	(	const float *	columns,
		int	channels,
		int	height,
		int	width,
		int	kH,
		int	kW,
		int	strideH,
		int	strideW,
		int	padH,
		int	padW,
		float *	outputImage )

inlinenoexcept

The adjoint of Im2Col(): scatters a (channels*kH*kW, outH*outW) column-gradient buffer back into a (channels, height, width) image.

Verified (see file-level note) to be the TRUE mathematical adjoint of Im2Col(), not just an inverse-shaped operation – this is what makes it safe to use for ConvPCLayer's feedback term.

Parameters

columns	Input buffer of shape (channelskHkW, outH*outW), typically a gradient produced by a GEMM against the same weight matrix used in the forward Im2Col() pass.
channels	Number of output-image channels.
height	Output image height (the original pre-Im2Col input height).
width	Output image width (the original pre-Im2Col input width).
kH	Kernel height.
kW	Kernel width.
strideH	Vertical stride.
strideW	Horizontal stride.
padH	Vertical zero-padding.
padW	Horizontal zero-padding.
outputImage	Output buffer of shape (channels, height, width). ACCUMULATED into (values are added, not overwritten) – the caller must zero this buffer first if a fresh result is wanted.

Warning: Not batch-aware – call once per batch item, with columns and outputImage offset to that item's slice.; Accumulates into outputImage rather than overwriting it; failing to memset the destination first will add this call's result on top of whatever was already there.

◆ ConvOutDim()

int Deep::ConvOutDim	(	int	inDim,
		int	kernel,
		int	stride,
		int	pad )

inlinenoexcept

Computes a single convolution output dimension (height or width) from its corresponding input dimension and conv parameters.

Parameters

inDim	Input dimension (height or width) before convolution.
kernel	Kernel size along this dimension.
stride	Stride along this dimension.
pad	Zero-padding applied to both sides of this dimension.

Returns: The resulting output dimension: (inDim + 2*pad - kernel) / stride + 1.

◆ CreateBackend()

std::unique_ptr< IComputeBackend > Deep::CreateBackend ( DeviceType device )

Factory function to create the appropriate backend.

◆ d_eSigmoid() [1/2]

void Deep::d_eSigmoid	(	float *	,
		size_t	,
		bool	)

inlinestaticnoexcept

See Sigmoid.h.

◆ d_eSigmoid() [2/2]

void Deep::d_eSigmoid	(	float *RESTRICT	x,
		const size_t	n,
		const bool	activated = false )

inlinestaticnoexcept

In-place Elliot sigmoid derivative.

Parameters

x	Array to derive in place.
n	Length of x.
activated	If true, x already holds e_sigmoid(x); if false, x holds the pre-activation value and e_sigmoid(x) is computed first.

◆ d_eSigmoidInto() [1/2]

void Deep::d_eSigmoidInto	(	float *RESTRICT	dst,
		const float *RESTRICT	src,
		const size_t	n )

inlinestaticnoexcept

Two-buffer, single-pass Elliot sigmoid derivative: reads src, writes the derivative directly into dst.

◆ d_eSigmoidInto() [2/2]

void Deep::d_eSigmoidInto	(	float *	RESTRICT,
		const float *	RESTRICT,
		size_t	)

inlinestaticnoexcept

See Sigmoid.h.

◆ dGelu() [1/2]

void Deep::dGelu	(	float *	,
		size_t	,
		bool	)

inlinestaticnoexcept

See Gelu.h.

◆ dGelu() [2/2]

void Deep::dGelu	(	float *RESTRICT	x,
		const size_t	n,
		const bool	activated )

inlinestaticnoexcept

In-place GELU derivative.

Parameters

x	Array to derive in place. Unlike dRelu/dTanh/dSigmoid, this always treats x as the PRE-activation value – the formula is recomputed from scratch regardless of activated, so calling this after gelu() has already run in place on the same buffer gives a wrong result.
n	Length of x.
activated	Unused; present only for signature consistency with DerivativeFn.

◆ dGeluInto() [1/2]

void Deep::dGeluInto	(	float *RESTRICT	dst,
		const float *RESTRICT	src,
		const size_t	n )

inlinestaticnoexcept

Two-buffer, single-pass GELU derivative: reads src (the pre-activation value), writes the derivative directly into dst.

◆ dGeluInto() [2/2]

void Deep::dGeluInto	(	float *	RESTRICT,
		const float *	RESTRICT,
		size_t	)

inlinestaticnoexcept

See Gelu.h.

◆ dLinear() [1/2]

void Deep::dLinear	(	float *	,
		size_t	,
		bool	)

inlinestaticnoexcept

See Linear.h.

◆ dLinear() [2/2]

void Deep::dLinear	(	float *	x,
		size_t	n,
		bool	activated = false )

inlinestaticnoexcept

Derivative of linear(): a constant 1 everywhere.

◆ dLinearInto() [1/2]

void Deep::dLinearInto	(	float *RESTRICT	dst,
		const float *RESTRICT	src,
		size_t	n )

inlinestaticnoexcept

Two-buffer variant of dLinear – src is unused (the derivative of a linear function is a constant), kept for signature consistency with To_dFn2's dispatch table.

◆ dLinearInto() [2/2]

void Deep::dLinearInto	(	float *	RESTRICT,
		const float *	RESTRICT,
		size_t	)

inlinestaticnoexcept

See Linear.h.

◆ dRelu() [1/2]

void Deep::dRelu	(	float *	,
		size_t	,
		bool	)

inlinestaticnoexcept

See Relu.h.

◆ dRelu() [2/2]

void Deep::dRelu	(	float *RESTRICT	x,
		const size_t	n,
		bool	activated = false )

inlinestaticnoexcept

In-place ReLU derivative: 1 where x > 0, else 0.

Parameters

x	Array to derive in place.
n	Length of x.
activated	Unused – ReLU's derivative only depends on sign, so whether x already holds the activated value makes no difference. Present only for signature consistency with DerivativeFn.

◆ dReluInto() [1/2]

void Deep::dReluInto	(	float *RESTRICT	dst,
		const float *RESTRICT	src,
		const size_t	n )

inlinestaticnoexcept

Two-buffer variant of dRelu: reads src, writes the derivative directly into dst.

◆ dReluInto() [2/2]

void Deep::dReluInto	(	float *	RESTRICT,
		const float *	RESTRICT,
		size_t	)

inlinestaticnoexcept

See Relu.h.

◆ dSigmoid() [1/2]

void Deep::dSigmoid	(	float *	,
		size_t	,
		bool	)

inlinestaticnoexcept

See Sigmoid.h.

◆ dSigmoid() [2/2]

void Deep::dSigmoid	(	float *RESTRICT	x,
		const size_t	n,
		bool	activated = false )

inlinestaticnoexcept

In-place logistic sigmoid derivative: sigmoid(x)*(1-sigmoid(x)).

Parameters

x	Array to derive in place.
n	Length of x.
activated	If true, x already holds sigmoid(x); if false, x holds the pre-activation value and sigmoid(x) is computed first.

◆ dSigmoidInto() [1/2]

void Deep::dSigmoidInto	(	float *RESTRICT	dst,
		const float *RESTRICT	src,
		const size_t	n )

inlinestaticnoexcept

Two-buffer, single-pass sigmoid derivative: reads src, writes sigmoid(src)*(1-sigmoid(src)) directly into dst. Computes the sigmoid into a register and derives from that same register immediately – one read of src, one write to dst, no intermediate array pass.

◆ dSigmoidInto() [2/2]

void Deep::dSigmoidInto	(	float *	RESTRICT,
		const float *	RESTRICT,
		size_t	)

inlinestaticnoexcept

See Sigmoid.h.

◆ dTanh() [1/2]

void Deep::dTanh	(	float *	,
		size_t	,
		bool	)

inlinestaticnoexcept

See Tanh.h.

◆ dTanh() [2/2]

void Deep::dTanh	(	float *RESTRICT	x,
		const size_t	n,
		const bool	activated = false )

inlinestaticnoexcept

In-place tanh derivative: 1 - tanh(x)^2.

Parameters

x	Array to derive in place.
n	Length of x.
activated	If true, x already holds tanh(x) (skips re-computing it); if false, x holds the pre-activation value and tanh(x) is computed first.

◆ dTanhInto() [1/2]

void Deep::dTanhInto	(	float *RESTRICT	dst,
		const float *RESTRICT	src,
		const size_t	n )

inlinestaticnoexcept

Two-buffer, single-pass tanh derivative: reads src, writes 1-tanh(src)^2 directly into dst. Unlike dTanh(x, n, false), which needs two full passes internally (activate in place, then derive in place) plus a separate copy beforehand if src can't be mutated, this computes tanh(src[i]) into a register and derives from that same register immediately, with only one read of src and one write to dst – no intermediate array pass at all.

◆ dTanhInto() [2/2]

void Deep::dTanhInto	(	float *	RESTRICT,
		const float *	RESTRICT,
		size_t	)

inlinestaticnoexcept

See Tanh.h.

◆ DynamicThread()

void Deep::DynamicThread ( int batchSize )

inlinestaticnoexcept

Sets OpenMP's (and the BLAS library's) thread count for the given batch size, memoized so it's a no-op if the target hasn't changed since the last call. Small batches get 1 thread (avoids parallelization overhead dominating tiny work); larger batches scale up to a measured, capped optimum – see the inline note on why 16 threads, not the full core count, is the ceiling.

Parameters

batchSize The batch size about to be processed.

◆ e_sigmoid() [1/2]

void Deep::e_sigmoid	(	float *	,
		size_t	)

inlinestaticnoexcept

See Sigmoid.h.

◆ e_sigmoid() [2/2]

void Deep::e_sigmoid	(	float *RESTRICT	x,
		const size_t	n )

inlinenoexcept

Implements the Elliot Sigmoid approximation, i.e. S(x) = (1/2)((x / (1 + |x|)) + 1).

Parameters

x	array, does not need to be aligned
n	x length

◆ expf_v()

void Deep::expf_v	(	float *	x,
		size_t	n )

inlinestatic

In-place vectorized exp: x[i] = exp(x[i]) for all i.

◆ gelu() [1/2]

void Deep::gelu	(	float *	,
		size_t	)

inlinestaticnoexcept

See Gelu.h.

◆ gelu() [2/2]

void Deep::gelu	(	float *RESTRICT	x,
		const size_t	n )

inlinestaticnoexcept

In-place GELU activation (tanh approximation, ~5e-4 max absolute error vs. true GELU – see tests/tGelu.cpp).

◆ GetL2CacheBytes()

size_t Deep::GetL2CacheBytes ( )

inline

Queries the size of this CPU's L2 cache (per-core, level 2), falling back to 512 KiB on platforms where it can't be queried.

◆ hsum128_ps()

float Deep::hsum128_ps ( __m128 x )

inlinestatic

Horizontal sum of all 4 lanes of an SSE register.

◆ hsum256_ps()

float Deep::hsum256_ps ( __m256 x )

inlinestatic

Horizontal sum of all 8 lanes of an AVX2 register.

◆ Im2Col()

void Deep::Im2Col	(	const float *	input,
		int	channels,
		int	height,
		int	width,
		int	kH,
		int	kW,
		int	strideH,
		int	strideW,
		int	padH,
		int	padW,
		float *	columns )

inlinenoexcept

Rearranges a single (channels, height, width) input image into a (channels*kH*kW, outH*outW) column matrix, where each column holds one flattened receptive-field patch – the standard im2col transform that lets convolution be computed as a single GEMM against a (outChannels, channels*kH*kW) weight matrix.

Positions that fall outside the input (due to padding) are written as zero.

Parameters

input	Pointer to a single batch item's input image, layout (channels, height, width), row-major, contiguous per channel.
channels	Number of input channels.
height	Input image height.
width	Input image width.
kH	Kernel height.
kW	Kernel width.
strideH	Vertical stride.
strideW	Horizontal stride.
padH	Vertical zero-padding.
padW	Horizontal zero-padding.
columns	Output buffer of shape (channelskHkW, outH*outW), fully overwritten (not accumulated into). Must be preallocated by the caller with size computed from ConvOutDim().

Warning: Not batch-aware – call once per batch item, with input and columns offset to that item's slice.

◆ LayerProfile()

template<typename LayerT>

ProfileAccumulator & Deep::LayerProfile ( LayerT * layer )

inline

Looks up (or creates, on first call) layer's accumulator in the process-wide registry, keyed by its address.

Template Parameters

LayerT Any layer type exposing GetInputSize()/GetOutputSize().

◆ linear() [1/2]

void Deep::linear	(	float *	,
		size_t	)

inlinestaticnoexcept

See Linear.h.

◆ linear() [2/2]

void Deep::linear	(	float *	x,
		size_t	n )

inlinestaticnoexcept

Identity activation: f(x) = x. A no-op, kept as a real function so it can be used interchangeably with the other activations through ActivationFn.

◆ logf_v()

void Deep::logf_v	(	float *	x,
		size_t	n )

inline

In-place vectorized log: x[i] = log(x[i]) for all i.

◆ PrintAllProfiles()

void Deep::PrintAllProfiles ( double wallClockSeconds )

inline

Prints every registered layer's profile, then a grand-total reconciliation against an independently measured wall-clock time.

Parameters

wallClockSeconds Wall-clock duration of the whole run, measured separately (e.g. std::chrono around the training loop) – NOT derived from these accumulators.

◆ ProfileRegistry()

std::vector< std::pair< const void *, ProfileEntry > > & Deep::ProfileRegistry ( )

inline

The process-wide table of every layer that's had LayerProfile() called on it at least once, in first-call order.

◆ ProtectFPU()

void Deep::ProtectFPU ( )

inlinestatic

Flushes denormal floats to zero on both input and output of SSE FP ops, avoiding the severe slowdown x86 FPUs hit computing with denormals – settling can drive values arbitrarily close to zero over many steps.

◆ relu() [1/2]

void Deep::relu	(	float *	,
		size_t	)

inlinestaticnoexcept

See Relu.h.

◆ relu() [2/2]

void Deep::relu	(	float *RESTRICT	x,
		const size_t	n )

inlinestaticnoexcept

RELU(x) = MAX(0, x) for all x.

Parameters

x	array
n	x length

◆ sigmoid() [1/2]

void Deep::sigmoid	(	float *	,
		size_t	)

inlinestaticnoexcept

See Sigmoid.h.

◆ sigmoid() [2/2]

void Deep::sigmoid	(	float *RESTRICT	x,
		const size_t	n )

inlinestaticnoexcept

Implements the Logistic Sigmoid approximation, i.e. S(x) = 1 / (1 + e^(-x)).

Parameters

x	array, does not need to be aligned
n	x length

◆ tanh() [1/2]

void Deep::tanh	(	float *	,
		size_t	)

inlinestaticnoexcept

See Tanh.h.

◆ tanh() [2/2]

void Deep::tanh	(	float *RESTRICT	x,
		const size_t	n )

inlinestaticnoexcept

In-place hyperbolic tangent activation.

Parameters

x	Array to activate in place.
n	Length of x.

◆ To_AType() [1/2]

ActivationType Deep::To_AType ( ActivationFn fn )

inlinestatic

Reverse lookup of To_Fn(): maps a function pointer back to its ActivationType, by identity comparison against every known activation function.

Returns: ActivationType::NONE if fn doesn't match any known activation.

◆ To_AType() [2/2]

ActivationType Deep::To_AType ( DerivativeFn dfn )

inlinestatic

Same as the ActivationFn overload above, for derivative function pointers.

Returns: ActivationType::NONE if dfn doesn't match any known derivative.

◆ To_dFn()

DerivativeFn Deep::To_dFn ( ActivationType dType )

inlinestatic

Maps an ActivationType (a derivative variant, e.g. dRELU) to its concrete function pointer.

Returns: nullptr for ActivationType::NONE or an unrecognized value.

◆ To_dFn2()

DerivativeFn2 Deep::To_dFn2 ( ActivationType dType )

inlinestatic

Dispatches to the two-buffer (dst, src, n) derivative variant for the given activation type – see DerivativeFn2.

◆ To_Fn()

ActivationFn Deep::To_Fn ( ActivationType type )

inlinestatic

Maps an ActivationType to its concrete function pointer.

Returns: nullptr for ActivationType::NONE or an unrecognized value.

◆ toOpType()

OptimizerType Deep::toOpType ( const char * s )

inline

Parses an OptimizerType from its name.

Parameters

s	Null-terminated string; expected to start with "SGD", "ADAM", or "ADAMW" (checked in that order, so "ADAMW" must be tested before a bare "ADAM" prefix match would otherwise win).

Returns: The matching OptimizerType, or OptimizerType::SGD if s doesn't match any recognized name.

Variable Documentation

◆ MAGIC_GELU_1

float Deep::MAGIC_GELU_1 = 0.7978845608028654f

constexpr

sqrt(2/pi), the leading constant in the tanh approximation of GELU. Also used directly by CUDABackend's device kernels.

◆ MAGIC_GELU_2

float Deep::MAGIC_GELU_2 = 0.044715f

constexpr

The tanh approximation's cubic-term coefficient (0.044715), paired with MAGIC_GELU_1.

Namespaces

Classes

Typedefs

Enumerations

Functions

Variables

Detailed Description

Typedef Documentation

◆ ActivationFn

◆ DerivativeFn

◆ DerivativeFn2

Enumeration Type Documentation

◆ ActivationType

◆ DeviceType

◆ OptimizerType

Function Documentation

◆ AdamUpdate()

◆ AdamWUpdate()

◆ AutoBatchSize()

◆ Col2Im()

◆ ConvOutDim()

◆ CreateBackend()

◆ d_eSigmoid() [1/2]

◆ d_eSigmoid() [2/2]

◆ d_eSigmoidInto() [1/2]

◆ d_eSigmoidInto() [2/2]

◆ dGelu() [1/2]

◆ dGelu() [2/2]

◆ dGeluInto() [1/2]

◆ dGeluInto() [2/2]

◆ dLinear() [1/2]

◆ dLinear() [2/2]

◆ dLinearInto() [1/2]

◆ dLinearInto() [2/2]

◆ dRelu() [1/2]

◆ dRelu() [2/2]

◆ dReluInto() [1/2]

◆ dReluInto() [2/2]

◆ dSigmoid() [1/2]

◆ dSigmoid() [2/2]

◆ dSigmoidInto() [1/2]

◆ dSigmoidInto() [2/2]

◆ dTanh() [1/2]

◆ dTanh() [2/2]

◆ dTanhInto() [1/2]

◆ dTanhInto() [2/2]

◆ DynamicThread()

◆ e_sigmoid() [1/2]

◆ e_sigmoid() [2/2]

◆ expf_v()

◆ gelu() [1/2]

◆ gelu() [2/2]

◆ GetL2CacheBytes()

◆ hsum128_ps()

◆ hsum256_ps()

◆ Im2Col()

◆ LayerProfile()

◆ linear() [1/2]

◆ linear() [2/2]

◆ logf_v()

◆ PrintAllProfiles()

◆ ProfileRegistry()

◆ ProtectFPU()

◆ relu() [1/2]

◆ relu() [2/2]

◆ sigmoid() [1/2]

◆ sigmoid() [2/2]

◆ tanh() [1/2]

◆ tanh() [2/2]

◆ To_AType() [1/2]

◆ To_AType() [2/2]

◆ To_dFn()

◆ To_dFn2()

◆ To_Fn()

◆ toOpType()

Variable Documentation

◆ MAGIC_GELU_1

◆ MAGIC_GELU_2