Enter – a modern solution designed to handle the messy, heterogeneous reality of advanced computing. The Problem with Traditional Batched GEMM Imagine training a recommendation system with embedding tables of varying sizes, or running inference on a transformer model with variable sequence lengths. In these scenarios, you might have 1,024 independent GEMM operations, each with different M, N, or K dimensions.
cublasLtGroupedMatmulPlan_t groupPlans[3]; for (int i = 0; i < groupCount; i++) { cublasLtGroupedMatmulPlanInit(handle, matmulDesc, &groupPlans[i], CUDA_R_16F, CUDA_R_16F, CUDA_R_16F, CUDA_R_32F, m_arr[i], n, k); } cublaslt grouped gemm
cublasLtMatmulDesc_t matmulDesc; cublasLtMatmulDescCreate(&matmulDesc, CUDA_R_32F, CUDA_R_16F); Enter – a modern solution designed to handle
// Allocate and fill matrices...
Traditional cuBLAS offers batched GEMM (e.g., cublas<t>gemmBatched ), which runs a list of independent matrix multiplications. However, it comes with a major limitation: (M, N, K) and data types. cublasLtGroupedMatmulPlan_t groupPlans[3]; for (int i = 0; i
In the world of High-Performance Computing (HPC) and Deep Learning (DL), the General Matrix Multiply (GEMM) operation is the undisputed king. From large language models (LLMs) to scientific simulations, performance often hinges on how efficiently you can compute C = α*A*B + β*C .
float alpha = 1.0f, beta = 0.0f; cublasLtMatmulGrouped(handle, nullptr, matmulDesc, &alpha, &beta, (void**)A_ptrs, (void**)B_ptrs, (void**)C_ptrs, (void**)C_ptrs, groupCount, groupPlans); cuBLASLt Grouped GEMM represents a paradigm shift for batched linear algebra on GPUs. It acknowledges that real-world workloads are irregular, heterogeneous, and dynamic. By moving the complexity of scheduling and fusing into the library, it allows developers to write clean, expressive code that still achieves near-peak hardware performance.