Attention without softmax: the experiment that challenges LLMs

Today’s most important AI news

An unusually technical Sunday: the news dominating today comes from the open-source community, not the big labs. The central topic is architectural — someone has just published a model that removes one of the most fundamental pieces of modern transformers. At the same time, interest is resurfacing in making AI more accessible for those who want to build, not just consume. A day for curious developers.

A model without softmax: can you throw out a fundamental piece of the transformer?

Since transformers became AI’s dominant architecture, the softmax function has been an immovable piece of the attention mechanism. It calculates how much “weight” the model gives each token when processing a sequence. Nobody questioned it much — it worked, period. That logic just got a serious push: an independent researcher under the alias NonGameCatharsis published this week a 354-million-parameter model trained on 11.5 billion tokens that replaces softmax with structural sparsity and custom Triton kernels to skip unnecessary blocks of computation.

The most immediate practical result is VRAM savings on long contexts. One of the real bottlenecks when working with extended context windows isn’t just compute time — it’s GPU memory. Standard attention has a quadratic memory cost with respect to context length. The tile-skipping this model proposes attacks exactly that point: instead of computing attention over every pair of tokens, it identifies and skips the blocks that contribute little to the final result.

The weights are publicly available along with the Triton kernel code. This isn’t closed-lab research — any team can download the model today and reproduce the experiments. At this scale (GPT-2 Medium), the conclusions can’t be extrapolated directly to models with billions of parameters, but the precedent is valuable: it shows the attention architecture has more room for exploration than market consensus suggests.

For teams working with long documents, RAG over extensive knowledge bases or any application where context is a scarce resource, this work is worth following closely. If the techniques scale, they could translate into cheaper inference without sacrificing quality.

Building an LLM from scratch: a resource for those who learn by doing

While papers advance, the gap between “understanding LLMs conceptually” and “knowing how to build one” remains huge for most technology professionals. This week a complete workshop titled “Build Your Own LLM” appeared on YouTube that tries to close that gap with an uncommon approach: it doesn’t require math or machine learning knowledge as a starting point.

The material covers the full path — from perceptrons and activation functions (ReLU, GELU, SwiGLU) to the transformer architecture, through training and post-training. What sets it apart from similar resources is the teaching method: it uses executable code and Excel examples to build intuition before introducing formalism. For a product manager or developer who works with AI APIs but has never touched PyTorch, that entry point completely changes accessibility.

It’s not a resource for those who want to do cutting-edge research. It’s for those who want to stop treating LLMs as black boxes and start reasoning about their limitations, biases and possibilities more precisely. In a market where a new model appears every week with claims that are hard to evaluate, that kind of architectural understanding has direct value in product and engineering decisions.

Dirty data in time series: the silent problem that ruins models

A story closer to data teams’ everyday work: a developer published TSAuditor, an auditing framework for time series, after getting burned by a classic problem. The developer had ten years of historical data, the profiling tool reported 3% missing data — within tolerance — and yet the downstream models behaved badly. The real problem wasn’t the amount of missing data but its temporal distribution: the gaps were concentrated in specific periods, which created artificial patterns the model learned as if they were real signal.

This type of error is especially treacherous in time series because generic validation tools aren’t designed to detect temporal dependencies. 3% missing values distributed randomly is manageable noise. The same 3% concentrated on the Mondays of a specific year may be an operational anomaly that contaminates any forecasting model.

TSAuditor aims to make exactly that kind of problem visible before it reaches production. For teams working with financial, operations or demand data, or any signal with temporal structure, the tool deserves a look. The cost of detecting these problems late — after a forecasting model has been in production for weeks — is considerably higher than auditing properly from the start.

In a nutshell

What ties today’s news together is a pattern repeating with growing force in 2026: real progress doesn’t always come from the labs with the most compute. It comes from people asking uncomfortable questions about pieces that “always worked this way” — like softmax — or solving concrete problems the big tools ignore because they’re too specific. The open-source community is operating with a speed and honesty about limitations that corporate launches rarely have. For professionals deciding which technology to adopt, that’s increasingly relevant: the map of the territory is no longer drawn only by the big companies.

---

Sources used: https://www.reddit.com/r/MachineLearning/comments/1ubmybr/i_released_a_softmaxfree_attention_model_at_gpt2/, https://www.reddit.com/r/MachineLearning/comments/1uazlnd/hi_reddit_i_posted_my_build_your_own_llm_workshop/, https://www.reddit.com/r/MachineLearning/comments/1ub15wf/tsauditor_a_timeseries_auditing_framework_p/