Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation
Computer Science > Machine Learning arXiv:2602.00294 (cs) [Submitted on 30 Jan 2026] Title:Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation Authors:Franz A. Heinsen, Leo Kozachkov View a PDF of the paper titled Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation, by Franz A. Heinsen and Leo Kozachkov View PDF HTML (experimental) Abstract:The most widely used artificial intelligence (AI) models today are Transformers employing self-attention. In its standard form, self-attention incurs costs that increase with context length, driving demand for storage, compute, and energy that is now outstripping society’s ability to provide them. To help address this issue, we show that self-attention is efficiently computable…
