One Person, One Algorithm, and the Weight of a World-Changing Choice

In the hypothetical scenario where an individual discovers or invents an algorithm for true AI or AGI, the decision on how to proceed is fraught with ethical implications.

3 min readMachine Learning

The scenario this reader lays out is not hypothetical in the way it first appears. It is a thought experiment with a timer on it, and the fact that we are having it in a public forum means we are already late to the conversation. One person, alone, holding the key to true artificial general intelligence, forced to decide whether to hand it to a corporation, a government, the public, or no one at all. That is not a puzzle. That is a pressure test for every value we claim to hold.

Our take is simple: the choice is not about the algorithm. It is about the illusion that any single choice is safe. Open source means everyone gets it, including the people who will use it to build surveillance states or autonomous weapons. A research paper is just open source with extra steps, and no one should pretend otherwise. A tech company is not a neutral actor; it is a profit engine, and handing over AGI to a profit engine is not a decision, it is a surrender. The government option is not better, because governments are not in the business of wisdom, they are in the business of power. And keeping it secret means betting that no one else ever has the same bad day, which is not a bet anyone should take seriously.

So what does that leave? It leaves the uncomfortable truth that this person, whoever they are, is not solving a technical problem. They are solving a trust problem, and they have no good options because trust was never the issue. The issue is that we have built systems where the only choices are between bad actors, and that is not a failure of the algorithm. It is a failure of our collective imagination. We have spent decades assuming that technology will save us, and now we are faced with the possibility that the person holding the keys is just as lost as the rest of us.

The practical takeaway for our readers is not to wait for that person to appear. It is to recognize that the dilemma is already here, in smaller forms, every time a developer decides whether to release a dataset, every time a researcher weighs the risk of publishing a paper, every time a company decides what to do with the models they have trained. The choice is not coming. It is happening. And the only way to prepare for it is to stop pretending that there is a right answer and start building the kind of institutions, norms, and safeguards that make the wrong answer less catastrophic. That is not a slogan. That is the work.

From Machine Learning

So, let's say some random guy in the middle of nowhere finds or invents an algorithm leading to true AI* / AGI. What should that guy even do?

Making it open source would let absolutely anyone make AI. Of course, "anyone" includes both the most altruistic people on Earth and psychopaths hellbent on causing misery on others.

Read the original at Machine Learning