Witchcraft, fast local semantic search on top of SQLite [P]
Our take
The recent development of Witchcraft, an open-source project from Dropbox, marks an exciting leap in the realm of semantic search technology. By re-implementing Stanford's XTR-Warp search engine in safe Rust and utilizing a single-file SQLite database for client-side deployment, Witchcraft not only streamlines the search process but also enhances accessibility for users. This project eliminates the need for API keys, vector databases, or complex chunking strategies, boasting impressive performance metrics with a mere 20ms end-to-end search latency. Such advancements resonate with ongoing conversations in the AI and machine learning communities, as seen in discussions around initiatives like MLRC 2026 is open for submissions - an official track at NeurIPS 2026 and the ethical implications of AI technologies explored in AI/ML Ethicists.
Witchcraft's design philosophy prioritizes user experience, making it easy for individuals to implement powerful semantic search capabilities directly on their devices. This stands in stark contrast to traditional systems that often require substantial infrastructure or cloud services. The inclusion of Pickbrain, a command-line interface that indexes user sessions for rapid retrieval, further exemplifies the human-centered approach of this project. This feature allows users to efficiently rediscover previous conversations and easily transition between projects, reinforcing productivity. As tools like Pickbrain become integral to our workflow, they empower users to leverage their data in meaningful ways, transforming the mundane task of searching into a seamless experience.
The broader significance of Witchcraft lies in its potential to democratize access to advanced semantic search technology. By removing barriers such as dependency on external databases or API services, this project invites a wider audience to engage with sophisticated data management practices. This mirrors trends seen in other areas of AI, such as the increasing focus on reproducibility highlighted by events like the Machine Learning Reproducibility Challenge. The ability for developers and data scientists to experiment with and refine their tools independently not only fosters innovation but also cultivates a community of practice that can collaboratively push the boundaries of what is possible in AI-driven data management.
Looking ahead, the implications of Witchcraft extend beyond mere efficiency gains. As we witness a shift towards more localized, user-controlled data solutions, questions arise about the future of data privacy and autonomy. Will innovations like Witchcraft inspire a movement away from cloud dependency, allowing users to maintain more control over their information? As semantic search continues to evolve, it will be crucial for the community to navigate these developments thoughtfully, ensuring that the tools we create not only enhance productivity but also respect user agency. As we embrace these transformative solutions, it is worth considering how we can collectively shape a future where data management is not just efficient but also equitable and user-friendly.
Witchcraft (https://github.com/dropbox/witchcraft), an open source project that I built at Dropbox, is a from-scratch re-implementation of Stanford's XTR-Warp semantic search engine ( https://github.com/jlscheerer/xtr-warp ) in safe rust, using a single-file SQLite database as backing storage, making it suitable for client-side deployment. It runs completely stand-alone on your device, needs no API keys, no vector database, no chunking strategy, no fancy re-rankers, and it is lightning fast (20ms p.95 end-to-end search latency on NFCorpus, at 33% NDCG@10, on an Apple Macbook Pro M2 Max, more than twice as fast as the original XTR-WARP on server-class hardware, at similar accuracy.)
The project also includes Pickbrain, a CLI that indexes your Claude Code and OpenAI Codex session transcripts, memory files, and authored documents into a Witchcraft database for fast semantic search. Ever wondered "what was that conversation where I fixed the auth middleware?" — pickbrain finds it, and lets you resume the session directly. There is also a /pickbrain skill for both Claude and Codex, which equips those tools with global memory across all sessions. You can use pickbrain directly from the command line, e.g., to rediscover a previous agent session and directly resume it, or you can have your agent invoke it via the supplied skill, e.g.,. "use /pickbrain to read up on our previous efforts on training with XTR token masking", to easily populate a new session with previous context.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience