Gelu.h File Reference

GELU (tanh approximation) and its derivative (plain and two-buffer variants), with AVX512/AVX2/SSE paths and a scalar tail. More...

#include <cassert>
#include <cstddef>
#include <immintrin.h>
#include <sleef.h>
#include <deepity/utils/activations/ActivationMacros.h>

Go to the source code of this file.

Namespaces
namespace	Deep
	This file exists to provide both Tensor.h and IComputeBackend.h with device types without introducing circular dependencies.

Functions
static void	Deep::gelu (float *RESTRICT x, const size_t n) noexcept
	In-place GELU activation (tanh approximation, ~5e-4 max absolute error vs. true GELU – see tests/tGelu.cpp).
static void	Deep::dGelu (float *RESTRICT x, const size_t n, const bool activated) noexcept
	In-place GELU derivative.
static void	Deep::dGeluInto (float *RESTRICT dst, const float *RESTRICT src, const size_t n) noexcept
	Two-buffer, single-pass GELU derivative: reads src (the pre-activation value), writes the derivative directly into dst.

Variables
constexpr float	Deep::MAGIC_GELU_1 = 0.7978845608028654f
	sqrt(2/pi), the leading constant in the tanh approximation of GELU. Also used directly by CUDABackend's device kernels.
constexpr float	Deep::MAGIC_GELU_2 = 0.044715f
	The tanh approximation's cubic-term coefficient (0.044715), paired with MAGIC_GELU_1.

Detailed Description

GELU (tanh approximation) and its derivative (plain and two-buffer variants), with AVX512/AVX2/SSE paths and a scalar tail.