**Our Take: The Hybrid Fine-Tuning Frontier Isn't a One-Day Project, But Someone Needs to Map It**
This post from a self-taught developer charting a course into Nemotron 3 Nano fine-tuning is exactly the kind of work that moves the field forward, and our opinion is that they're asking the right questions, even if the answers aren't yet written. The choice of a 30B-A3B hybrid Mamba-Attention-MoE architecture for multi-task reasoning is structurally sound. Mamba-2 layers map cleanly to state-aware comprehension across context, and the sparse MoE design with 128 experts and top-6 routing offers capacity without per-token compute blowup. But the gap between architectural potential and practical fine-tuning is real, and the user has identified the precise points where standard tutorials fall silent.
The router under LoRA question is the most immediate practical concern. Standard dense-transformer LoRA assumes attention projections are the primary adaptation surface. In a hybrid MoE, the router determines which experts activate, and freezing it means the model can only adapt *within* the existing expert assignments rather than learning *which* experts to route to for each task. This is not a trivial distinction. If multi-task specialization depends on expert-level separation, freezing the router may push all tasks into overlapping expert sets, defeating the architecture's purpose. Conversely, LoRA-ing the router introduces gradient dynamics that are poorly documented at this scale. The auxiliary load-balancing loss compounds the uncertainty: uneven example counts across four capabilities could fight that loss rather than work with it, forcing experts toward uniform activation at the expense of task-specific specialization.
The Mamba-2 layer adaptation concern is equally significant. Selective SSM states have different stability properties than attention. Low-rank perturbation on input/output projections could shift state initialization behavior in ways that affect recurrence, especially over longer contexts. Standard rank sweet spots of 8-32 come from attention-based work; whether they hold for SSM projection structure is an open question. Using held-out per-capability eval sets, built and frozen before training, is the right defense against the silent degradation risk. In sparse MoE, a single capability can degrade while aggregate metrics look fine if different experts handle different tasks. That cannot be caught with a single benchmark.
What this means for anyone attempting similar work is that the first run will provide negative results worth more than a hundred blog posts. Document the router gradient behavior, track expert utilization per capability across training, and compare frozen vs. LoRA-ed router outcomes even if the latter fails. The same applies to the Mamba-2 layer projections: log stability metrics and compare to attention-only baselines. The $120 budget across 5-6 iterations on a rented H100 is tight but realistic for probing these dimensions. The user will likely discover that load-balancing loss fights multi-task specialization more than expected, and that a single round of per-capability held-out eval catches collapses that loss curves miss entirely. That is not failure. It is the data that the field needs but has not yet produced.