Attention

Attention methods control how query, key, and value tensors are combined during inference
Attention is important because it can be one of the most expensive parts of a model, strongly affecting generation speed and memory use
SD.Next can use the selected implementation directly or route compatible operations through faster specialized kernels, falling back automatically when a method does not support the device, tensor shape, dtype, or mask

Attention methods

Available in settings -> cross attention -> attention method

Scaled-Dot-Product

PyTorch's built-in scaled dot-product attention (SDPA), providing a broadly compatible default implementation
See https://docs.pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html for more information

Dynamic attention

SD.Next's dynamic SDPA implementation, which adapts attention computation and slicing to the available device and memory.

SDNQ attention

SDNQ's optimized attention kernel, designed for supported long sequences and configurable quantized matrix operations
Written in Triton and providing broad GPU support
Requires functional triton and build tools

xFormers

xFormers' memory-efficient attention implementation
Requires the xformers package and compatible model support
See https://github.com/facebookresearch/xformers for more information and installation instructions

Flex attention

PyTorch Flex Attention, which supports programmable score and block-mask logic for dense or sparse attention patterns

Flash attention

The FlashAttention kernel, which computes attention efficiently on supported GPUs without materializing the full attention matrix
Requires functional flash-attn package
See https://github.com/Dao-AILab/flash-attention for more information and installation instructions

Sage attention

SageAttention's quantized attention kernels, optimized for supported GPU architectures and head dimensions
Requires functional sage-attention package
See https://github.com/thu-ml/SageAttention for more information and installation instructions

Triton AMD Flash attention

An SD.Next's AMD-focused FlashAttention implementation built with Triton and available on ROCm and compatible ZLUDA setups
Requires functional triton and build tools

Sparse attention

Sparse attention reduces computation by selecting only a subset of query-key blocks instead of evaluating the full attention matrix
It is applied during transformer self-attention when the sequence is long enough and the selected backend supports block masks; cross-attention, unsupported operations, and excluded models remain dense automatically

Can be combined with any attention method for additional memory savings

Attention dispatcher

Attention dispatcher can be used to select attention method that is not available as built-in in SD.Next by using kernels package that allows download and registration of custom pre-compiled attention kernels
See https://huggingface.co/docs/kernels/index for more information