Crazy model! It actually uses the old Qwen2.5-Coder-3B stack and got really great performance with their post-training stack. Need to use it in the ne...

Cool new open-weight model by Cohere: a new lightweight 30B open-weight model for agentic coding tasks. This one builds on Command A+ using the parall...

Turns out Fable 5 is shadowbanning AI researchers 🫤

Always back to the basics: LatentMoE was probably inspired by MLA, which was inspired by LoRA, which was inspired by SVD, which was inspired by eigend...

http://x.com/i/article/2063647807437705216

And another open-weight release. Nemotron 3 Ultra has an ultra impressive capability:efficiency ratio! Design-wise, it carries forward the Mamba-2-att...

It's been a while! 4 nice additions to the open-weight local-LLM-on-consumer-hardware ecosystem:

The MiniMax M2 series was one of the most widely used open-weight LLM series earlier this year. Now, we got a technical report with some interesting t...

Added a DeepSeek Sparse Attention (DSA) from-scratch implementation to my LLMs-from-scratch repo thanks to an awesome new reader contrib. With motivat...

It's been *almost* a bit quiet around LLM architecture releases in the past two weeks 😅 Interesting tidbit is the parallel block design. Via the Cm...

New article: a visual tour of recent LLM architecture advances, from Gemma 4 to DeepSeek V4. I focus on long-context efficiency tweaks like KV sharing...

A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch. And how I approach new open-weight models, ...

Interesting paper. What I like about this is that it is a relatively low-commitment attention modification. I.e., one can use it during most of traini...

Back from a little family break! Lots has happened, and I’m planning to do a deeper dive into the most interesting architectural components (soon). B...

Had a great time at PyCon & PyData DE. Highly recommend it. Great open-source, community-focused conference with lots of builders in the Python AI, LL...

RT Sten Rüdiger: I’ve uploaded a new paper on arXiv (co-authored by @rasbt): MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning In Parameter-...

Added an RSS feed to the LLM Architecture Gallery so it is a bit easier to keep up with new additions over time: https://sebastianraschka.com/llm-arch...

Components of a coding agent: a little write-up on the building blocks behind coding agents, from repo context and tool use to memory and delegation. ...

http://x.com/i/article/2038978163389112321

RT Liquidity Goblin: In an effort to try stop seeing so much slop I've been trying to train my own AI detection model. Found something incredibly inte...

RT levi: Day 83/365 of GPU Programming Looking at DeepSeek's Multi-Head Latent Attention today. The last part of the AMD challenge series is to optimi...

A visual guide to modern LLM attention variants, all in one place: https://magazine.sebastianraschka.com/p/visual-attention-variants

RT Sebastian Raschka: Re The Redbubble poster version just arrived (https://www.redbubble.com/i/poster/LLM-Architecture-Gallery-by-Ahead-of-AI/1792744...

Oh wow, Mamba-3 is here! For me, the most interesting use case of Mamba and Mamba-likes are the recent transformer attention hybrid architectures (Qwe...