The question of whether a commit was written by a human or an AI assistant is becoming a fixture in modern software delivery, and the approach outlined here, anchored in Git metadata, commit trailers, and diff patterns, is a pragmatic starting point. But the core problem is not a lack of signals; it is a crisis of calibration. A 500-line commit might be a junior developer scaffolding a feature, or it might be an AI generating boilerplate that a senior dev reviews and merges in minutes. The metadata that could disambiguate the two is often stripped away the moment code leaves the IDE. That is the real friction: provenance is fragile, and once it is gone, you are left with statistical guesswork.
What we find compelling is the shift in framing from binary classification to probabilistic risk scoring. Treating this as a confidence problem rather than a truth problem is the only honest approach. You cannot reliably say "this was AI-written," but you can say "this commit has a high probability of AI assistance based on these observable patterns." That is a useful distinction for engineering leaders who want visibility, not a police state. It aligns with the kind of practical evaluation we see elsewhere in the ecosystem, such as the work on Jev vs LLMs: Evaluating AI for Practical Decision-Making, where the focus is on calibration and confidence thresholds rather than absolute accuracy. Similarly, the conversation around Building an Internal Developer Platform with Artificial Intelligence suggests that agents are increasingly operating on semantic signals from Git and issue trackers, so why not apply the same logic in reverse to detect AI involvement?
The harder question is whether repository-level signals can ever be sufficient on their own. We would argue they cannot, and that is not a failure of effort but a property of the domain. A developer can always rewrite a commit message, split a large diff, or reorder hunks. That does not mean the exercise is pointless, it means you need to combine signals with earlier provenance capture. Instead of trying to reverse-engineer intent after the fact, instrument the development workflow to preserve it at the source. That could mean IDE plugins that add a structured trailer, or CI checks that require a signed attestation for AI-generated code. The Monitor Cypress Tests with Grafana: Persistent Observability for Your Data article demonstrates how persistent observability transforms ephemeral test results into durable insights; the same principle applies here, capture the signal where it is born, not where it decays.
Our take is straightforward: stop chasing a perfect detector and build a risk model with explicit, measurable trade-offs. Define your false-positive tolerance up front. If you are flagging 10% of human-written commits as AI-assisted, that is a cost you might accept for better coverage. If you are missing 40% of AI-generated code, that is a different problem. Calibration is a valid concern, but the answer is not a universal threshold, it is a policy decision tied to your team's risk appetite. The one concrete takeaway we would offer: start by tracking the false-positive rate on your own repository history before you ever ship a detection rule to production. Because once you start flagging commits, you are not just measuring the code, you are changing the behavior of the people who write it. What happens to that dynamic is the open question worth watching.