A new arXiv paper introduces Δ-Mem, a compact fixed-size memory mechanism that augments frozen LLMs with delta-rule learning. It achieves 1.31× improvement on MemoryAgentBench using just an 8×8 state matrix, without retraining the base model.
A project post on r/MachineLearning stood out because it did not just propose an alternative attention score; it documented the engineering breakage that follows when dot products disappear.
The March 20, 2026 HN discussion around Attention Residuals focused on a simple claim with large implications: replace fixed residual addition with learned depth-wise attention and recover performance with modest overhead.
A Reddit thread surfaced Kimi's AttnRes paper, which argues that fixed residual accumulation in PreNorm LLMs dilutes deeper layers. The proposed attention-based residual path and its block variant aim to keep the gains without exploding memory cost.
A popular r/MachineLearning discussion examines an unofficial theorem-style claim that Attention’s core optimization geometry is d^2, not n^2. Community response is mixed: strong curiosity, but equally strong calls for peer review and reproducible evidence.