May 7, 2026•3 min read•from Machine Learning

I trained a NER model on 33,000 Indian Supreme Court judgments (1950–2024) CASE_CITATION hits 97.76% F1, +17 points over the only prior baseline [P]

Our take

Introducing en_legal_ner_ind_trf v0.1, a fine-tuned NER model based on InLegalBERT, trained on approximately 33,000 Indian Supreme Court judgments from 1950 to 2024. Achieving an impressive 97.76% F1 score for CASE_CITATION—17 points higher than the existing OpenNyAI model—this release marks a significant advancement in legal data processing. With 13 labels and an overall F1 of 78.67%, it addresses the limitations of prior models, especially with pre-1990 texts.

The release of en_legal_ner_ind_trf v0.1 represents more than just another academic exercise in legal natural language processing—it signals a maturation of India's homegrown legal AI ecosystem. When we consider the broader landscape of [20M+ Indian legal documents with citation graphs and vector embeddings – potential uses for legal NLP? [D]](post/20m-indian-legal-documents-with-citation-graphs-and-vector-e-cmnyxkj0g0f5dzxsxktu1x4bw), this new model emerges as a critical infrastructure component that transforms raw legal text into structured, actionable intelligence. The fact that CASE_CITATION hits 97.76% F1 while outperforming OpenNyAI by 17 percentage points suggests we're witnessing a meaningful leap forward in how machines can parse and understand India's complex legal corpus.

What makes this particularly compelling is the strategic focus on India's foundational legal documents—the first four decades of Supreme Court judgments that have historically suffered from OCR degradation and limited computational analysis. While 20M+ Indian legal documents provide the raw material, models like this give us the tools to actually extract meaning from that vast archive. The silver annotation pipeline, combining regex patterns, metadata projection, transformer-based labeling, and gazetteer matching, demonstrates a pragmatic approach to scaling legal NLP without requiring massive manual annotation efforts.

The performance breakdown tells an honest story about where legal NER stands today. CASE_CITATION and PROVISION scores above 96% show that well-defined, structured entities are increasingly within reach of automated systems. However, the lower scores for GPE, ORG, and PETITIONER reveal the persistent challenge of context-dependent classification in legal text—where "State of Maharashtra" might be a petitioner in one case and merely a geographical reference in another. This isn't a failure of the model so much as an acknowledgment that legal language operates on multiple semantic levels simultaneously, requiring more sophisticated architectures than simple token classification.

Looking ahead, the planned v1.0 release with CRF head integration and gold annotation validation promises to address several current limitations. The positional bias issues and pre-1990 OCR noise problems highlight how legal AI must grapple with historical digitization challenges that don't exist in cleaner domains. As we move toward more robust legal intelligence systems, the question becomes whether the next generation of models will successfully integrate document-level context, temporal reasoning, and institutional role classification into unified frameworks. The intersection of 20M+ Indian legal documents with increasingly sophisticated NER models suggests we're approaching a tipping point where legal research, precedent analysis, and case outcome prediction could become genuinely accessible tools rather than specialized expert functions.

TL;DR: Released en_legal_ner_ind_trf v0.1 - InLegalBERT fine-tuned on ~34,700 silver-annotated chunks from 33k Indian SC judgments. 13 labels. 78.67% overall F1. CASE_CITATION at 97.76% already exceeds OpenNyAI's PRECEDENT score by +17 points. Free, Apache-2.0.

Why this exists

OpenNyAI is the only prior Indian legal NER model with any community presence. It's unmaintained and degrades on pre-1990 OCR-era text - the first 40 years of India's constitutional jurisprudence.

No replacement existed.

Results

Entity	F1	Support
CASE_CITATION	97.76%	3,821
PROVISION	96.35%	20,248
STATUTE	91.94%	8,187
LAWYER	74.67%	3,982
JUDGE	68.06%	1,978
DATE	55.15%	3,289
RESPONDENT	50.44%	1,731
COURT	50.34%	1,033
WITNESS	49.77%	762
OTHER_PERSON	47.11%	4,266
PETITIONER	44.71%	1,573
ORG	41.34%	2,128
GPE	36.56% ⚠	1,197
micro avg	78.67%	54,195

Evaluated on a held-out validation split (~500 documents, stride=512, non-overlapping). The 25-file locked test set is untouched - head-to-head with OpenNyAI runs in v1.0.

Comparison note: OpenNyAI (RoBERTa + transition-based parser, gold-annotated) achieved 91.1% overall strict F1. Not directly comparable - different test sets, different annotation quality, different corpus scope. The +17 point gap on CASE_CITATION is the one apples-to-apples number worth flagging.

The annotation pipeline

Silver labels from four automatic pipelines merged per document:

Regex — 14-pattern citation extractor + statute/provision extractor → CASE_CITATION, STATUTE, PROVISION
Metadata projection — case metadata JSONs mapped to character offsets via RapidFuzz → JUDGE, PETITIONER, RESPONDENT
Transformer NER — OpenNyAI en_legal_ner_trf, offset-corrected → LAWYER, COURT, ORG, GPE, DATE, OTHER_PERSON, WITNESS
Gazetteer — 858 Central Acts with alias resolution → confirms and adds STATUTE spans

Trained with Focal Loss (γ=2.0) to handle label imbalance between STATUTE/CASE_CITATION and O tokens. Hardware: Kaggle T4 (free tier).

Known weak spots - being honest

GPE (36.56%) and ORG (41.34%) are the problem labels. In Indian legal text, "State of Maharashtra" or "Union of India" appear as GPE, PETITIONER, RESPONDENT, or ORG depending on context. A linear token classification head can't resolve overlapping roles. CRF head is v1.0's job.

Positional bias - silver training data has repetitive header structures. Performance degrades when parties appear mid-document.

Pre-1990 OCR noise - judgments from 1950–1989 vary in quality. Recall drops the further back you go.

What's next

300-file gold annotation is in progress (3 volunteers onboard). v1.0 will add a CRF head, run the locked test set, and publish the official head-to-head with OpenNyAI.

Model: huggingface.co/evolawyer/inlegalbert-sc-ner-silver

Dataset: huggingface.co/datasets/evolawyer/indian-sc-judgments-ner-silver

GitHub: github.com/evolawyer/inlegalbert-sc-ner-silver

Happy to go deep on the annotation pipeline, conflict resolution between the four label sources, or the Focal Loss setup.

submitted by /u/gkv856
[link] [comments]

Read on the original site

Open the publisher's page for the full experience

View original article →

20M+ Indian legal documents with citation graphs and vector embeddings – potential uses for legal NLP? [D]been working on structuring India's legal corpus for the past 2 years and wanted to share what I've built and hear from people working on legal NLP or low-resource Indian language models. dataset is 20M+ Indian court cases from the Supreme Court, all 25 High Courts, and 14 Tribunals. each case has structured metadata (court, bench, date, parties, judges, sections cited, acts referenced, case type). there's a citation graph across the full corpus where I've classified relationships as followed, distinguished, overruled, or mentioned. every case is embedded with Voyage AI (1024d dense) plus BM25 sparse vectors. I have also cross-referenced 23,122 Acts and Statutes with the cases that interpret them. Some things that might be interesting to this community: citation network thing across 20M+ cases is, as far as I know, the first machine-readable one for Indian law. could be useful for graph neural network research, legal outcome prediction, or influence analysis on which judgments are most cited and which are being overruled. most Indian language NLP corpora are conversational or news text. Legal text is a completely different register. formal, precise, domain-specific. the bilingual pairs from the translation service could be useful for fine-tuning Indian language models on formal and legal domains. the metadata extraction pipeline identifies judges, advocates, parties, sections, acts, and dates from unstructured judgment text. built with a mix of regex, heuristics, and LLM-based extraction. the structured outputs could serve as training data for legal NER models. Indian court judgments are long. Median around 3,000 words, some exceed 50,000 words. if anyone is benchmarking retrieval-augmented generation on legal domains, this corpus plus the citation graph could work as an evaluation bed. Ground truth exists in the citation relationships: if Case A cites Case B, a good retriever should show B when asked about the legal question in A. data is available via API and bulk export in JSON and Parquet. Indian court judgments are public domain under Indian law so no copyright issues for research use. being upfront about limitations: coverage is primarily English text (except Supreme court one, they have 3-4 translated language copies ) since Indian HCs issue orders in English, the regional language data comes from our translation service not from original regional language judgments. metadata extraction accuracy varies by court, SC and major HCs are cleaner while smaller tribunals have messier inputs. The citation graph is extracted heuristically plus LLM-assisted, I estimate around 90-95% precision on citation extraction and lower on treatment classification. Not all 20M cases have complete metadata, coverage is best for post-2007 judgments. would love to hear from anyone working on legal NLP, Indian language models, or graph-based legal analysis. What would be most useful to you from a dataset like this? deets at vaquill submitted by /u/zriyansh [link] [comments]

I trained a NER model on 33,000 Indian Supreme Court judgments (1950–2024) CASE_CITATION hits 97.76% F1, +17 points over the only prior baseline [P]

Related Articles