Continued development of the model based on the SSN [D]
Our take
The resurgence of Project NORD, specifically the shift towards NORD 5.5 – Flash, highlights a fascinating, albeit niche, exploration within the AI landscape. It’s encouraging to see developers pushing beyond the dominant Transformer architecture, particularly with a focus on CPU-first inference. This contrasts with much of the current momentum, which prioritizes GPU acceleration. The developer’s decision to rebuild rather than simply patching the existing architecture speaks to a deeper conviction about fundamental design choices. It echoes recent developments like Spotify's work on external indexing for data lakes [Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake], demonstrating a broader trend of optimizing for efficiency and accessibility, rather than solely pursuing raw computational power. Furthermore, the focus on simplifying the core architecture, moving away from artificial spike-time dimensions and embracing the language sequence as the time axis, suggests a desire for greater elegance and clarity in the model's structure – a welcome departure from the increasingly complex layers often seen in modern AI.
The project’s commitment to a CPU-first approach is particularly noteworthy. While GPUs have undeniably propelled advancements in language modeling, the cost and accessibility barriers associated with them are significant. NORD 5.5’s design choices—causal convolution-style token mixing, token-time LIF/event dynamics, and a top-1 sparse MoE—all point towards a model that’s intentionally engineered for efficient CPU execution. This aligns with the growing conversation around democratizing AI and making it more accessible to a wider range of users and hardware. The developer's acknowledgement that this is “still very much an experiment” is refreshing; it avoids the hype often surrounding AI advancements and frames the work as a genuine exploration. The comparison planned between NORD 5.0 and 5.5, across various parameters like RAM usage and perplexity, will provide valuable data for understanding the impact of these architectural changes, and is particularly relevant given the ongoing concerns around AI governance and potential failures in agent context layers [Agent context layers: Enterprises governing their AI data are catching twice as many bad answers as the ones who aren't].
Beyond the specific technical details, NORD 5.5's approach raises broader questions about the future of AI model design. The move away from a purely "brain-inspired" approach towards a focus on functional coherence—making the remaining components work together properly—is a crucial insight. It suggests that simply layering on more biologically-inspired elements doesn’t guarantee improved performance; rather, a holistic design that prioritizes integration and efficiency is essential. The developer’s openness to feedback from researchers working on SNNs, recurrent models, and sparse MoE further underscores the collaborative nature of this exploration. This contrasts with some corners of the AI community that exhibit a tendency towards proprietary development, and aligns with the broader industry discussion about the potential of stateless systems [MCP Goes Stateless, and Developers Ask Whether That Just Makes It an API Again].
Ultimately, while NORD 5.5 is unlikely to displace established architectures like Transformers anytime soon, its focus on CPU-first inference, simplified design, and commitment to experimentation represents a valuable contribution to the AI landscape. The results of the planned benchmarking will be closely watched, particularly by those seeking more accessible and efficient AI solutions. The key question now is whether this approach can yield meaningful performance gains while maintaining a manageable level of complexity, and whether this CPU-centric focus will attract further development and community support.
Back after ~6 months — rebuilding my spiking language model around CPU-first inference
Hey everyone. It’s been around six months since I last posted anything about this project here.
Some of you might remember Project NORD, my experimental hybrid spiking / brain-inspired language model architecture. I basicall disappeared for a while 😅, but recently I came back to the project, went through the old architecture again, and realized I didn’t really want to keep stacking fixes on top of it. So instead, I’ve started rebuilding a pretty large part of the system. The new version is called:
NORD 5.5 — Flash The main idea this time is pretty simple:
What happens if I design the architecture around CPU inference from the beginning, instead of building soething Transformer-like and trying to optimize it later? A lot is changing internally. The current design uses things like: strictly causal processing no standard quadratic attention in the main inference path causal convolution-style token mixing token-time LIF / event dynamics sensory → association → memory → executive processing stages top-1 sparse MoE + a shared expert persistent recurrent memory separate structural, personal and auxiliary memory banks
persistent recurrent identity state factorized vocabulary embedding/output streaming token-by-token inference One of the biggest changes is actually something much simpler. Older versions of NORD used an artificial internal spike-time dimension, roughly like this: token -> T0 -> T1 -> T2 -> ... -> T9 I’m mostly getting rid of that. Instead, the actual language sequence becomes the time axis: token0 -> token1 -> token2 -> token3 -> ... That removes a lot of intermediate state and makes the whole architecture considerably cleaner. Going back through the old code also exposed a few things I wasn’t very happy with. Some experimental modules weren’t completely causal, memory was coupled too much to sequence shape, and parts of the STDP system ended up being more disconnected from real training than I originally intended. So NORD 5.5 isn’t really about throwing even more “brain-inspired” components into the model. It’s mostly about simplifying the core and making the things that remain actually work together properly. I’m definitely not claiming this is going to beat Transformers, RWKV-style models, linear attention models, etc. Right now it’s still very much an experiment. The part that actually matters comes next: training and benchmarking it.
Things I want to compare: NORD 5.0 vs NORD 5.5 CPU tokens/sec RAM usage perplexity / validation loss long-context behaviour memory on/off MoE on/off spiking components on/off I’m especially curious to hear from anyone working on SNNs, recurrent models, sparse MoE, CPU inference, or weird alternative language-model architectures in general.
After not touching the project seriously for about half a year, it feels surprisingly good to be building it again 😅
I’ll post actual numbers once I have something that’s worth benchmarking instead of just architecture diagrams.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience