Deepity: A C++ library showing Predictive Coding Networks can match Backprop (97.73% on MNIST in 60s) [P]

The recent emergence of Deepity, a C++ machine learning library focused on Predictive Coding Networks (PCNs), represents a significant, albeit quietly impactful, development in the pursuit of biologically plausible and potentially more robust AI architectures. While the broader machine learning community remains largely captivated by transformer models and their derivatives, the ongoing exploration of alternative credit assignment algorithms – the mechanisms by which neural networks learn – is crucial for long-term progress. This effort, as highlighted in a recent Reddit post, demonstrates a tangible step towards closing the performance gap between PCNs and established backpropagation methods, achieving a remarkable 97.73% test accuracy on MNIST in just 59.5 seconds, a figure competitive with PyTorch’s backpropagation implementation. The work echoes the spirit of innovation seen in other areas of applied machine learning, such as Swiggy’s development of an in house predicted lifetime value model using more than 350 pre order features and a multi task MLP [Swiggy Uses 350+ Features and Multi-Task MLP to Predict Customer Lifetime Value]. Furthermore, the discussion around the continued utility of Hidden Markov Models for unsupervised tasks [Are HMMs still used for unsupervised tasks?] underscores a broader interest in exploring alternative approaches to data exploration and understanding, a theme deeply aligned with the goals of PCN research.

The core challenge with PCNs has always been their computational efficiency. While theoretically appealing due to their resemblance to how the brain is believed to learn, naive implementations historically lagged far behind backpropagation in terms of speed. The Deepity project’s success hinges on two key innovations: the incorporation of recent research on Accelerated PCNs via Direct Kolen-Pollack Feedback Alignment, and the implementation of algorithmic caching to minimize redundant computations during inference. This combination allows Deepity to achieve competitive performance without sacrificing the inherent benefits of PCNs, such as their potential for continual learning – a capability that standard backpropagation often struggles with. The decision to build this library in C++ is also noteworthy, suggesting a focus on performance and control, allowing for fine-grained optimization that might be more difficult in higher-level frameworks. This stands in contrast to the sometimes opaque nature of larger, more complex AI ecosystems, offering a glimpse into a more granular understanding and manipulation of the underlying algorithms.

The implications of this work extend beyond MNIST classification. The ability to train PCNs efficiently opens doors to exploring their potential in more complex and dynamic environments. The project’s stated next steps – porting the kernels to CUDA for GPU acceleration and testing its capabilities in continual learning scenarios – are particularly exciting. Continual learning, the ability for a model to learn new tasks without forgetting previously learned ones, remains a significant hurdle for current AI systems, and PCNs offer a compelling alternative to backpropagation in this domain. The open-source nature of Deepity, with its accessible GitHub repository and project site, encourages community involvement and collaboration, further accelerating its development and potential impact. This mirrors a trend toward decentralized AI development, moving away from solely relying on the resources of large tech companies.

Ultimately, Deepity’s progress highlights the value of persistent exploration of alternative machine learning paradigms. While backpropagation has served as the workhorse of deep learning for decades, it’s crucial to investigate alternative credit assignment mechanisms to unlock new capabilities and address the limitations of current approaches. The question now becomes: will this focus on local learning and alternative credit assignment yield breakthroughs in areas where traditional backpropagation falls short, particularly in domains requiring adaptability, robustness, and a deeper understanding of the underlying data structures? The continued development of Deepity, and similar projects, is a space worth watching closely.