The instinct behind this question bank is the right one: past papers are the most honest measure of what a school actually teaches, and turning them into a living assessment resource respects the curriculum instead of fighting it. The approach here, using the curriculum's learning objectives as a guard rail, requiring admin approval, and keeping exam content on the school's own server, shows a maturity that's rare in edtech. The 80% pass rate for multiple-choice questions is a solid starting point, not a failure. The real signal is in what the builder admits: structured and practical questions are weaker, and the small model invents wrong chemistry in lab setups. That's not a bug to paper over; it's the exact place where human oversight matters most.
The decision to check answer keys in code rather than trusting the model is the detail worth copying. Too many AI-assisted tools treat model output as ground truth, and that's how subtle errors slip into classrooms. This builder understands that the model is a draft generator, not an authority. The same logic should extend to the PDF extraction problem. Sending whole PDFs to an LLM works, but it's fragile, science notation gets mangled, tables interleave, and the model has to guess at structure. The question of whether docling or marker actually improves precision for equations and diagrams is still open, and that's a practical bottleneck. If the extraction layer can't preserve chemistry notation reliably, the downstream question quality will always be capped, regardless of how good the generation model is.
On retrieval, the instinct to start with objective tags is sound. Semantic search via pgvector is tempting, but the bank is small and the curriculum objectives already give precise, human-meaningful categories. Adding vector search from day one would add operational complexity without a clear payoff. The smarter path is to wait until the bank grows large enough that duplicate or near-duplicate questions become a real problem, then introduce semantic de-duplication as a targeted fix. Cost at scale is the other open question. Batch APIs and smaller models can handle thousands of questions, but the quality fall-off in structured items is where the budget should go. The mid-size model fixed most of the chemistry errors, which suggests a tiered approach, small model for routine generation, mid-size for high-stakes or practical questions, is the pragmatic route.
What this project really needs is a better eval framework. Human review is necessary but not sufficient. The blind-answer check is clever, but it only catches internal inconsistency, not whether a question actually measures the learning objective it claims to tag. A meaningful eval would track how students perform on generated questions versus past-paper originals, and whether the generated items discriminate between mastery levels. That kind of longitudinal data is the only way to know if the bank is genuinely useful, not just technically correct. The builder should watch for that signal early, even if it means starting with a small pilot cohort. The promise here is real, but the discipline to measure learning outcomes, not just generation accuracy, will determine whether this becomes a trusted tool or another AI demo that teachers quietly ignore.