DharmaOCR: Open-Source Specialized SLM (3B) + Cost–Performance Benchmark against LLMs and other open-sourced models [R]
Our take
Hey everyone, we just open-sourced DharmaOCR on Hugging Face. Models and datasets are all public, free to use and experiment with.
We also published the paper documenting all the experimentation behind it, for those who want to dig into the methodology.
We fine-tuned open-source SLMs (3B and 7B parameters) using SFT + DPO and ran them against GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.6, Google Document AI, and open-source alternatives like OlmOCR, Deepseek-OCR, GLMOCR, and Qwen3.
- The specialized models came out on top: 0.925 (7B) and 0.911 (3B).
- DPO using the model's own degenerate outputs as rejected examples cut the failure rate by 87.6%.
- AWQ quantization drops per-page inference cost ~22%, with insignificant effect on performance.
Models & datasets: https://huggingface.co/Dharma-AI
Full paper: https://arxiv.org/abs/2604.14314
Paper summary: https://gist.science/paper/2604.14314
[link] [comments]
Read on the original site
Open the publisher's page for the full experience
Related Articles
- We benchmarked 18 LLMs on OCR (7k+ calls) — cheaper/old models oftentimes win. Full dataset + framework open-sourced. [R]TLDR; We were overpaying for OCR, so we compared flagship models with cheaper and older models. New mini-bench + leaderboard. Free tool to test your own documents. Open Source. We’ve been looking at OCR / document extraction workflows and kept seeing the same pattern: Too many teams are either stuck in legacy OCR pipelines, or are overpaying badly for LLM calls by defaulting to the newest/ biggest model. We put together a curated set of 42 standard documents and ran every model 10 times under identical conditions; 7,560 total calls. Main takeaway: for standard OCR, smaller and older models match premium accuracy at a fraction of the cost. We track pass^n (reliability at scale), cost-per-success, latency, and critical field accuracy. Everything is open source: https://github.com/ArbitrHq/ocr-mini-bench Leaderboard: https://arbitrhq.ai/leaderboards/ Curious whether this matches what others here are seeing. submitted by /u/TimoKerre [link] [comments]
- We open-sourced Chaperone-Thinking-LQ-1.0 — a 4-bit GPTQ + QLoRA fine-tuned DeepSeek-R1-32B that hits 84% on MedQA in ~20GB[N]Hey everyone, We just open-sourced our reasoning model, Chaperone-Thinking-LQ-1.0, on Hugging Face. It's built on DeepSeek-R1-Distill-Qwen-32B but goes well beyond a simple quantization — here's what we actually did: The pipeline: 4-bit GPTQ quantization — compressed the model from ~60GB down to ~20GB Quantization-aware training (QAT) via GPTQ with calibration to minimize accuracy loss QLoRA fine-tuning on medical and scientific corpora Removed the adaptive identity layer for transparency — the model correctly attributes its architecture to DeepSeek's original work Results: Benchmark Chaperone-Thinking-LQ-1.0 DeepSeek-R1 OpenAI-o1-1217 MATH-500 91.9 97.3 96.4 MMLU 85.9 90.8 91.8 AIME 2024 66.7 79.8 79.2 GPQA Diamond 56.7 71.5 75.7 MedQA 84% — — MedQA is the headline — 84% accuracy, within 4 points of GPT-4o (~88%), in a model that fits on a single L40/L40s GPU. Speed: 36.86 tok/s throughput vs 22.84 tok/s for the base DeepSeek-R1-32B — about 1.6x faster with ~43% lower median latency. Why we did it: We needed a reasoning model that could run on-prem for enterprise healthcare clients with strict data sovereignty requirements. No API calls to OpenAI, no data leaving the building. Turns out, with the right optimization pipeline, you can get pretty close to frontier performance at a fraction of the cost. Download: https://huggingface.co/empirischtech/DeepSeek-R1-Distill-Qwen-32B-gptq-4bit License is CC-BY-4.0. Happy to answer questions about the pipeline, benchmarks, or deployment. submitted by /u/AltruisticCouple3491 [link] [comments]