Building a specialized legal AI model from scratch is the kind of project that reveals just how much we still don't know about teaching machines to reason. The developer behind this Polish legal model has done something most teams avoid: they've been honest about the confusion. After three rounds of continued pretraining on commentaries, legislation, and court decisions, they have a base model that performs slightly better than stock on a small quiz, and then the real questions begin. Should they keep feeding it data, or start supervised fine-tuning? How do you teach reasoning when your highest-quality Q&A comes without a chain of thought? And when does reinforcement learning actually enter the picture? These aren't just technical hurdles; they are the same tensions we see in other projects, like the model that A Model Promised Restraint Then Reached for Beijing: Raising Trust Questions, where a system's stated behavior and its actual reasoning diverged. The difference here is that this builder is trying to close that gap from the ground up.
The most practical insight from this story is about data hierarchy. The developer has access to Facebook legal Q&A groups, certified institutional answers, study materials, and thousands of harness conversations. The temptation is to throw everything into continued pretraining and hope the model absorbs the patterns. But that approach ignores a critical distinction: CPT teaches the model what language looks like in a domain, while SFT teaches it how to respond. Mixing messy social-media discussions with authoritative legislation during CPT risks diluting the signal, not strengthening it. A smarter path, one that aligns with what we've seen in Kimi K3.1 Pro Delivers Strong Coding and Reasoning at a Fraction of the Cost, is to reserve the highest-confidence, citation-backed Q&A for SFT, and use the rest of the data only after the model has a reliable chat baseline. Reasoning cannot be injected post-hoc; it must be demonstrated in the training examples themselves.
The developer's instinct to build a chatbot first, then add tool-use, is the right one. A model that cannot answer a legal question without searching for sources has not learned the law, it has learned how to search. The harness functions for vector and graph searches are powerful, but they should extend the model's knowledge, not replace it. Teaching the model to call those tools comes after it has developed a "hunch" about where to look, which is exactly what a well-structured SFT phase on expert Q&A can provide. The open question is how to handle the reasoning gap. As When a Small AI Model Follows a Planted Clue, Its Reasoning Shifts showed, even small models can be led astray by bad premises. For Polish law, where a wrong citation can have real consequences, the reward function in RL must penalize not just wrong conclusions, but unsupported reasoning paths. That is the hardest part of this project, and the one most worth watching.