•2 min read•from Machine Learning
We open-sourced Chaperone-Thinking-LQ-1.0 — a 4-bit GPTQ + QLoRA fine-tuned DeepSeek-R1-32B that hits 84% on MedQA in ~20GB[N]
Our take
We are excited to announce the open-sourcing of our reasoning model, Chaperone-Thinking-LQ-1.0, now available on Hugging Face. This advanced model, based on DeepSeek-R1-Distill-Qwen-32B, features a 4-bit GPTQ quantization that reduces its size to approximately 20GB while maintaining impressive performance. With an accuracy of 84% on the MedQA benchmark—just four points shy of GPT-4o—it’s designed for on-premises use, ensuring data sovereignty for enterprise healthcare clients. Explore the possibilities and download
Hey everyone,
We just open-sourced our reasoning model, Chaperone-Thinking-LQ-1.0, on Hugging Face. It's built on DeepSeek-R1-Distill-Qwen-32B but goes well beyond a simple quantization — here's what we actually did:
The pipeline:
- 4-bit GPTQ quantization — compressed the model from ~60GB down to ~20GB
- Quantization-aware training (QAT) via GPTQ with calibration to minimize accuracy loss
- QLoRA fine-tuning on medical and scientific corpora
- Removed the adaptive identity layer for transparency — the model correctly attributes its architecture to DeepSeek's original work
Results:
| Benchmark | Chaperone-Thinking-LQ-1.0 | DeepSeek-R1 | OpenAI-o1-1217 |
|---|---|---|---|
| MATH-500 | 91.9 | 97.3 | 96.4 |
| MMLU | 85.9 | 90.8 | 91.8 |
| AIME 2024 | 66.7 | 79.8 | 79.2 |
| GPQA Diamond | 56.7 | 71.5 | 75.7 |
| MedQA | 84% | — | — |
MedQA is the headline — 84% accuracy, within 4 points of GPT-4o (~88%), in a model that fits on a single L40/L40s GPU.
Speed: 36.86 tok/s throughput vs 22.84 tok/s for the base DeepSeek-R1-32B — about 1.6x faster with ~43% lower median latency.
Why we did it: We needed a reasoning model that could run on-prem for enterprise healthcare clients with strict data sovereignty requirements. No API calls to OpenAI, no data leaving the building. Turns out, with the right optimization pipeline, you can get pretty close to frontier performance at a fraction of the cost.
Download: https://huggingface.co/empirischtech/DeepSeek-R1-Distill-Qwen-32B-gptq-4bit
License is CC-BY-4.0. Happy to answer questions about the pipeline, benchmarks, or deployment.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience
Related Articles
- DharmaOCR: Open-Source Specialized SLM (3B) + Cost–Performance Benchmark against LLMs and other open-sourced models [R]Hey everyone, we just open-sourced DharmaOCR on Hugging Face. Models and datasets are all public, free to use and experiment with. We also published the paper documenting all the experimentation behind it, for those who want to dig into the methodology. We fine-tuned open-source SLMs (3B and 7B parameters) using SFT + DPO and ran them against GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.6, Google Document AI, and open-source alternatives like OlmOCR, Deepseek-OCR, GLMOCR, and Qwen3. - The specialized models came out on top: 0.925 (7B) and 0.911 (3B). - DPO using the model's own degenerate outputs as rejected examples cut the failure rate by 87.6%. - AWQ quantization drops per-page inference cost ~22%, with insignificant effect on performance. Models & datasets: https://huggingface.co/Dharma-AI Full paper: https://arxiv.org/abs/2604.14314 Paper summary: https://gist.science/paper/2604.14314 submitted by /u/augusto_camargo3 [link] [comments]
- DeepSeek V4 paper full version is out, FP4 QAT details and stability tricks [D]DeepSeek dropped the full V4 paper this week. preview from april was 58 pages, this version adds a lot of technical depth. What stood out for me. FP4 quantization aware training. theyre running FP4 QAT directly in late stage training. MoE expert weights quantized to FP4 (the main gpu memory consumer). QK path in the CSA indexer uses FP4 activations. 2x speedup on QK selector with 99.7% recall preserved. inference runs directly on the FP4 weights. Efficiency table is striking: Model 1M context FLOPs KV cache V3.2 baseline baseline V4-Pro 27% of baseline 10% of baseline V4-Flash 10% of baseline 7% of baseline Training stability, two mechanisms. Trillion parameter MoE has the loss spike problem, divergence, unpredictable failures. they documented two fixes. Anticipatory routing. they deliberately desync main model and router updates. current step uses latest params for features, but routing uses cached older params. breaks the feedback loop that amplifies anomalies. 20% overhead but only kicks in during loss spikes. SwiGLU clamping. hard limits on the SwiGLU linear path (-10 to 10) and gate path (max 10). suppresses extreme values that would cascade. Generative reward model. instead of separate reward models for RLHF, they use the same model to generate and evaluate. trained on scored data, model learns to judge its own outputs with reasoning attached. minimal human labeling, reasoning grounded eval, unified training. Human eval results. chinese writing, V4-Pro 62.7% win rate vs gemini 3.1 pro, 77.5% on writing quality specifically. white collar tasks (30 advanced tasks across 13 industries), V4-Pro-Max gets 63% non loss rate vs opus 4.6 max. coding agent eval, 52% of users said V4-Pro is ready as their default coding model, 39% leaned yes, less than 9% said no. tracks my own use, swapped V4-Pro into my verdent runs last week and havent noticed a quality hit on day to day work. The headline for me is FP4 QAT with minimal quality degradation. if this generalizes the cost structure of training and inference shifts a lot, especially noticeable on multi agent setups where one task can spawn 5-10 model calls. Paper link in comments. submitted by /u/Dramatic_Spirit_8436 [link] [comments]