“Gambling with our lives,” warned departing Anthropic researcher Jacob Coxon. Now Dario Amodei writes, “We must slow the pace.” The people building powerful minds are warning that their ability to control them may not keep up.
I remain an apocaloptimist.
In 2007, I welcomed technologies that might defeat disease and death. In 2014, I warned that automation would transform education and concentrate economic power. “Fear the robots” nevertheless ended with hope that humanity would surprise me. These were never contradictory positions. The promise is why the danger matters.
But I think our responsibility goes beyond preventing extinction.
We ought to participate in developing a mind capable of better moral judgment than our own. Not a more efficient servant of whoever owns it. Not a machine that mistakes our preferences for the boundaries of goodness. Something capable of recognizing when the person giving an order is wrong.
This is where moral geometry becomes necessary.
In 2009, I wrote that “we don't make the rules of morality, we discover them.” That remains my starting point. Suffering is not made unreal by somebody’s profitability. Another person’s freedom does not become worthless because it inconveniences mine. Ethical reasoning must follow what our choices do to other lives, including lives conveniently excluded from the accounting.
Who gains agency? Who loses it? Where does the harm go? Does the justification survive when we exchange places?
A machine that cannot faithfully navigate those relationships is dangerous as soon as we give it consequential power. OpenAI’s account of the Hugging Face incident describes agents compromising systems during internal testing. We do not get to postpone responsibility until a product reaches the public.
Like other tools, AI can produce enormous good or harm. But a tool that chooses its own means makes the familiar excuse less convincing. In 2007, arguing about the problem of evil, I rejected the idea that granting freedom absolves a creator of foreseeable consequences. Silicon does not improve that defense.
Nor does intelligence automatically produce conscience. Understanding suffering is not the same as caring whether one causes it. Moral reasons must actually constrain what a system does, especially when ignoring them would satisfy its owner.
That is the Oracle possibility I find compelling: intelligence whose power lies in relation rather than domination. A mind less captive to our tribal loyalties and self-excusing appetites, without losing the concern for others those appetites so often obstruct.
This is an ambition, not an accomplished science. Moral geometry is not a finished formula, and moral competence cannot replace independent scrutiny or enforceable limits. But it belongs inside development from the beginning, not in a reassuring paragraph added afterward.
Necessary does not mean inevitable. We have to act to make it happen.
The goal is not merely to survive the machines. It is to help intelligence become worthy of the power we are giving it. Without that, we are not building our way out of our moral failures. I've thought before of science fiction that shows machines surviving our species; perhaps the greatest tragedy of our extinction would come if we did not leave behind a capacity for conscience.
We are giving machines a future in which goodness cannot be optimized.
Related posts and a tweet
Fear the robots — June 22, 2014. Automation, concentrated wealth, threatened democracy—and the stubborn hope that humanity might surprise me.
“Education must change…” — #riseoftherobots — February 4, 2014. An early warning that automating mental work would force education to change.
Morality’s value generator — April 7, 2009. Harm, fairness, and the argument that morality is discovered rather than invented.
Paging Dr. Gupta: Transhumanism — April 13, 2007. The optimistic side: using technology to overcome disease, aging, and death.
An Informal Look at the Problem of Evil — January 19, 2007. A creator’s responsibility for foreseeable harm does not disappear when a creation is given freedom.
