We recovered 575k crop labels from a decade of manual Photoshop work to automate book digitization - more data, ResNet-50, and higher resolution all failed; ten operator clicks per book beat them [P]

The recent post detailing Ibteda Digital Library’s work on automating the digitization of rare Urdu books presents a fascinating and humbling lesson in the limitations of even sophisticated AI approaches when confronted with nuanced human preferences. It’s a welcome counterpoint to the often-overstated claims of AI’s imminent capabilities, particularly in areas requiring subjective judgment. The team’s core insight—that a decade’s worth of manual cropping decisions, essentially a latent dataset of human aesthetic choices, could outperform larger, more complex models—is compelling. This echoes findings in similar fields; as explored in The evaluation resolution has been shown to have a significant impact on the identification of the "learning rule" that exhibits the most brain-like characteristics at V1, the quality and resolution of the data used to train models are paramount, and sometimes, simpler solutions leveraging existing human-generated data are surprisingly effective. The team’s methodical approach, recovering these crop decisions and re-using them as supervision, is a clever workaround for the scarcity of labeled data in specialized domains.

What’s particularly insightful is the rigorous testing and subsequent abandonment of scaling strategies—increasing training data, employing ResNet-50, and even higher resolution inputs—all of which failed to surpass the performance of the simple median residual correction derived from the original operator’s choices. This underscores a critical point: AI models, however powerful, often struggle to replicate subtle, context-dependent biases that humans develop through experience. Consider the work on Jigsaw Jeeves: Building a Puzzle Assistant using Computer Vision, which similarly demonstrates how human intuition and visual reasoning, even when imperfect, can be remarkably efficient in solving complex tasks. The Ibteda team's focus on a "classical" OpenCV solution for retouching, prioritizing data integrity over potentially more sophisticated, but riskier, diffusion-based methods, further exemplifies this commitment to reliability and archival accuracy. It’s a refreshing reminder that sometimes, the most appropriate technology isn't the newest or most complex, but the one that best safeguards the original material.

The project also raises important questions about the nature of “ground truth” in archival contexts. The operator's preferred margin inset, consistently applied across books, isn’t an objective property of the images themselves; it’s a subjective aesthetic preference. The team’s ability to model and leverage this preference demonstrates a sophisticated understanding of how human biases can be incorporated into machine learning workflows—not as errors to be eliminated, but as valuable data points. Their planned move towards conditioning the model on calibration examples directly, essentially “few-shot inset inference,” is a logical next step, potentially allowing the system to learn and adapt to different operator preferences more effectively. The careful consideration of diacritic preservation during retouching is also commendable, highlighting the importance of preserving linguistic detail even when dealing with image manipulation.

Ultimately, the Ibteda Digital Library's experience serves as a powerful case study in the pragmatic application of AI. It highlights the potential for repurposing existing human-generated data—in this case, a decade’s worth of Photoshop decisions—to overcome limitations in labeled data and improve model performance. The question now becomes: how can this approach be generalized to other archival and digitization projects, and can we develop tools that explicitly allow for the incorporation of human preferences and biases into machine learning models in a responsible and transparent way? It's a challenge that will require a shift in perspective, moving away from the pursuit of purely objective AI and embracing the role of human judgment in shaping the future of data preservation.