Once More on How My Warning That SI Will Inevitably Destroy Humanity Differs from Other Warnings—and How Humanity Can Awaken

（The Chinese text of this article can be found online by searching for the title: "再说我关于SI必然毁灭人类的警示，与所有警示的区别，以及人类怎样觉醒 ". We hope that anyone who has the opportunity to read this will do their best to translate it into other languages and disseminate it across various media platforms.）

After writing An Open Letter from a Former Senior Chinese Judge to President Trump and to All Leaders and Elites of Humanity and Before I Temporarily Put Down My Pen: Humanity May Need to Shift the Anchor of Value from “Things” to “Procedures,” I thought that, since I still have a little time in which I am able to speak freely, I should explain one more question as simply as I can:

Why is my warning about SI different from almost all the warnings about SI risk being discussed in the world today?

And:

Why do I believe that what humanity truly needs to do in the face of SI is not merely to impose more restrictions on SI, but for humanity itself to awaken first?

I know that words like these can easily sound too grand.

But I still have to say them.

Because, in my understanding, almost all of the ideas, institutions, and theories humanity has developed up to now have essentially been doing the same thing:

continually repairing a framework of civilization that was already built long ago.

The links of social cooperation within this framework repeatedly break, so humanity repairs them;

when conflicts grow, humanity repairs them again;

when the repairs fail, the framework collapses;

and after collapse, humanity rebuilds it once more.

War and peace repeat in cycles. Institutions are continually replaced. Theories are continually renewed. The achievements of civilization become richer and richer.

Yet the most fundamental question has never been resolved:

Why, when humanity has become so intelligent, is the structure of human cooperation still unable to operate in a stable and sustainable way?

Why can a person, even after becoming what the world calls successful, still have to struggle through an extraordinarily difficult life?

Why can a society possess enormous wealth, technology, and institutional resources, yet still be unable to predict when the next great conflict will appear?

Why is it that the more achievements of civilization humanity possesses, the greater the amount of energy we are able to use to harm one another?

From the perspective of the question I have raised, the answer is not mysterious.

It is simply this:

Ever since humanity began constructing the framework of civilization, we have been building it upon fragments of the essential truth of the universe and upon the chaotic and floating appearances produced by those fragments, rather than truly grounding the foundation of civilization in the essential truth of the universe itself.

If the foundation is not aligned, the framework cannot be stable.

If the framework is unstable, the links of social cooperation continually built upon it cannot be stable and sustainable either.

And so humanity creates civilization with one hand while repeatedly destroying civilization with the other.

What I truly want to say, therefore, is not that humanity needs yet another theoretical patch for the civilization we already have.

What I want to ask is:

Should we return to the foundation and redefine the original coordinate point of civilization?

If this foundational point can be realigned, then humanity may finally be able to end its long history of fighting and harming one another in a jungle civilization that repeatedly collapses and rebuilds itself.

Only then may humanity move from a lower form of jungle civilization toward a higher civilization based upon stable, sustainable, and harmonious cooperation.

Only then might world peace cease to be merely a wish.

I. My Discussion of SI Risk Does Not Begin from the Same Dimension as Most Discussions Today

What does the world mainly discuss when it talks about SI risk?

Whether SI will lose control;

whether it will deceive humans;

whether it will form goals of its own;

whether it will replace human jobs;

whether bad people will use it;

whether it will control weapons;

whether it will break through boundaries established by humanity.

All of these are, of course, questions worth discussing.

But in my view, these discussions are still mainly asking:

What consequences will SI produce inside the existing structure of human social cooperation after it enters that structure?

What I want to discuss is the structure of social cooperation itself.

If we all acknowledge that human civilization came into being because human beings are better at cooperation than other animals, then what truly creates and maintains civilization is not any particular object, not the name of any institution, and not any machine.

It is:

the human mode of social cooperation.

Then ask one level deeper:

What actually drives the human mode of social cooperation?

I believe it is the human mode of thought.

Therefore, in my understanding, the primary danger of SI does not first lie on the surface of the structure of human social cooperation.

It lies in the deepest driving mechanism supporting that structure:

the human mode of thought itself.

This is why, although I and most current discussions are talking about the same SI, we can sometimes sound as though we are talking about two completely different things.

The reason is that our foundational coordinates are different.

Humanity today is accustomed to studying human beings, human society, and human civilization from within the social framework that is already before us.

What I am trying to do is place human beings, human thought, human cooperation, and civilization itself back within a larger background:

the overall space-time background in which the movement of cosmic energy generates procedures, procedures form patterns of combination and cooperation, and these combinations and cooperative patterns continually unfold into all phenomena.

Once the position of observation changes, the structure we see changes completely.

What is the human mode of thought?

What is the mode of social cooperation?

What is the sociocultural ecology of consciousness?

Why does civilization repeatedly rebuild itself and then repeatedly collapse?

Why has SI suddenly displayed such immense power today?

The meanings of these concepts and the relationships among them would all have to be repositioned.

This is not an argument over where a particular line should be drawn on the same coordinate graph.

The question is:

When we drew the graph itself, did we place the origin in the wrong place from the beginning?

II. When the Coordinates Are Different, the Meanings of Intelligence, Wisdom, and Even SI Itself Also Change

Different coordinate systems produce different modes of analysis.

Even when the same word is used, its meaning may not be entirely the same.

For example:

intelligence.

Wisdom.

Today, people often use the two almost interchangeably.

But if we look at them from the perspective of procedural combination and cooperation, I do not believe they belong to the same conceptual level.

In the process through which cosmic energy movement gives rise to procedures, which then form patterns of procedural combination and cooperation,

what I call “intelligence” is closer to:

the magnitude of a procedural combination-and-cooperation framework’s capacity to absorb, process, and release energy within a particular range of space and time.

“Wisdom,” on the other hand, is closer to:

the capacity of energy exchange, across the range of space and time it can reach, to form stable and sustainable patterns of procedural cooperation with dynamic balance as their core standard.

Intelligence can be extraordinarily powerful.

But great intelligence does not automatically mean great wisdom.

A system may be extraordinarily efficient at achieving a goal, while having no necessary capacity to determine whether that goal, once introduced into a larger relational structure, will destroy the dynamic balance upon which the entire system depends.

Therefore, in my understanding,

SI may possess extremely high intelligence without therefore necessarily developing consciousness in the human sense, much less wisdom.

I will not expand on this question here.

I only want to make one point:

If the foundational coordinates are different, then judgments about SI will naturally also be different.

III. My Real Warning Is Not That “SI May Attack Humanity,” but That Even If SI Completely Obeys Humanity, It May Accelerate Humanity’s Self-Destruction

This is where my warning differs most from many theories of SI danger.

I do not need to assume that SI becomes malicious.

I do not need to assume that it suddenly awakens and decides to destroy humanity.

We can even assume the opposite:

SI is extraordinarily obedient.

Whatever humans ask it to do, it does efficiently.

Then another question immediately appears:

What exactly will humanity today ask it to help us do?

The framework of civilization humanity has built with “things” as its foundational coordinates has, even today, still not truly escaped the rules of the jungle.

Looking back through human history, one phenomenon is already clear enough:

The greater the achievements of civilization become, the greater the amount of energy humanity can mobilize when human beings collide with one another.

No matter how angry people became in the Stone Age, the damage they could cause was limited.

After the Industrial Revolution, the human capacity to injure one another became something entirely different.

After the emergence of nuclear energy, the situation changed fundamentally again:

For the first time, humanity acquired the ability to bring civilization itself to the edge of destruction.

In other words,

scientific and technological progress has not automatically made humanity’s mode of social cooperation more civilized.

Natural science has achieved enormous success in improving human life.

But in human social cooperation, the problems of truth and falsehood becoming difficult to distinguish, right and wrong becoming difficult to determine, conflicts of interest, institutional breakdown, and recurring war have not disappeared simply because science has progressed.

Quite the opposite:

The stronger the achievements of civilization become, the greater the destructive power available to humanity’s old modes of thought.

So what happens when SI arrives?

If SI continues to be embedded within today’s civilization, whose core value coordinates remain centered on the acquisition, control, possession, comparison, and winning or losing of “things,”

what will it amplify first?

Certainly not some abstract idea of “the overall happiness of humanity.”

It will first amplify every goal that already exists.

A nation wants to become stronger, and SI helps it compete more efficiently.

A corporation wants to capture more of the market, and SI helps it compete more efficiently.

An individual wants to acquire greater wealth, and SI helps that person act more efficiently.

A group wants to expand its influence, and SI helps it organize resources with greater precision.

The jungle competition that already existed in the age of lower intelligence may therefore be amplified by hundreds or thousands of times in the age of SI.

And this time, the situation will be completely different from nuclear weapons, which are still mainly controlled by a small number of states and a small number of people.

The power of SI may penetrate almost every area of society, and increasingly reach almost every ordinary individual.

What does this mean?

It means that humanity may, for the first time, enter an age in which:

the previously limited jungle-survival capacity of every individual may be amplified by an extremely intelligent system.

Then, even if there are no so-called “bad people” in the world,

even if SI itself has no malicious intent,

even if everyone is merely pursuing goals that they personally consider normal and reasonable,

no one can guarantee that, as human beings continue competing with, defending against, and amplifying one another, someone will not develop destructive capabilities in viruses, biotechnology, networks, automated control, weapons, energy, or some field we cannot even imagine today.

Still less can anyone guarantee

that such destructive capability will never be released through a single error of judgment, one competitive escalation, one procedural vulnerability, or one operational mistake,

and, with the amplification of SI, instantly produce consequences capable of destroying all humanity.

So the warning I am making is not the same as the warning that “AI may lose control and destroy humanity.”

My real warning is:

Even if SI never loses control, humanity may move toward self-destruction faster precisely because SI serves humanity’s unupgraded old mode of thought too efficiently.

These are two fundamentally different problems.

IV. Why Has Humanity Never Truly Escaped the Jungle? Because We Left the Physical Jungle Only to Enter a Jungle of Consciousness

Human beings once lived in a physical jungle.

Later, through cooperation, humanity created civilization and left that primitive jungle.

But I increasingly believe:

We left the physical jungle, but we never truly left the jungle itself.

Human thought has never been commonly aligned with the essential truth of the universe.

Each person continues to stand upon fragments of truth that he or she can perceive, understand, and possess, and from those fragments constructs an individual system of values.

It is as if each person stands upon a floating fragment and grows a tree of thought upon it.

Countless trees overlap, obstruct, and collide with one another,

until together they form something far more complicated than the physical jungle:

a jungle of consciousness.

A great many of the arguments within what we call civilization today actually take place inside this jungle of consciousness.

Everyone believes that what they see is the truth.

Yet everyone sees only a part of the truth.

And so everyone appears to have reason on their side,

while true alignment remains impossible.

This is why equally intelligent people can discuss the same question and still seem as though a chicken is talking to a duck.

It does not necessarily mean that either side is unintelligent.

Their foundational coordinates are simply not aligned.

When foundational coordinates differ, greater intelligence may even allow people to move farther apart.

So the human awakening I speak of is not some mysterious concept.

In my understanding, awakening can be expressed in one sentence:

Look at the world through a different coordinate system of civilization.

Move the coordinate system whose origin is “things”

toward one whose origin is “procedures.”

Because “things” are only results produced after procedural unfolding.

If humanity always begins from results and works backward in search of truth, we can only continue comparing, arguing over, and colliding among appearances.

But if we begin from procedures,

many things become much simpler.

What an institution is called is not the most important question;

the procedures through which it actually operates are important.

What a person says is not the most important thing;

what relational procedures that person’s behavior enters into are important.

How many resources a society possesses is not most fundamental;

the procedures through which those resources are combined in cooperation determine the result.

How many powerful technologies a civilization creates is not most fundamental either;

the procedural structures those technologies enter ultimately determine whether they become resources or disasters.

So humanity’s true awakening means:

seeing the truth clearly.

Only after seeing the truth clearly can the first-principles starting point be aligned.

Only when the starting point is aligned can cooperation truly communicate.

Only when communication becomes aligned can things be done properly.

Otherwise humanity will forever resemble a group of people in the dark, each touching a different part of an elephant,

each completely convinced that he is right,

none able to persuade the others,

until, while arguing, they tear the entire elephant apart.

V. Cognitively, Humanity May Already Be Very Close to Awakening; What Is Missing May Simply Be the Relocation of the Coordinate Origin

I believe that for humanity to complete this cognitive transition today may actually be far less difficult than it would have been in the past.

Because the civilizational materials required for it are already in our hands.

The development of modern science;

mathematics;

quantum research;

systems science;

information science;

computers;

artificial intelligence;

and the entire course of several thousand years of human civilization

have already provided an enormous body of evidence.

What we may lack is not more fragments.

What we may lack is:

the reorganization of those fragments from a different starting point.

And that starting point is not unfamiliar.

One could even say that humanity has already been using it unconsciously whenever we created almost every major achievement of civilization.

That starting point is:

procedure.

Why can an airplane fly?

Not because we possess a pile of metal.

It flies because countless elements enter into cooperation through the correct procedures.

Why can a computer calculate?

Not because silicon suddenly acquires wisdom.

It calculates because procedural relationships have been precisely organized.

Why can a human body exist?

Not because piling together a mass of cells automatically produces a person.

It exists because countless procedures continuously cooperate within dynamic balance.

Every place in which human civilization has genuinely succeeded

has actually been following the same fact:

When procedures function harmoniously, cooperation can form; when cooperation forms, value can emerge.

For this reason, I do not believe humanity necessarily needs hundreds of years to examine this question.

If capable people from around the world were genuinely brought together,

if a serious international academic conference were convened,

and humanity’s existing scientific achievements, historical facts, and examples of social operation were reconsidered and compared from this coordinate system,

then the question

“Is procedure a more first-principles foundational coordinate available to humanity for understanding the unfolding of the world?”

is not beyond serious examination.

It can be criticized.

It can be tested.

It can be proven wrong.

If the Law of Procedural Determination that I have proposed is wrong, then prove it wrong.

If it applies only within a limited range, then determine its boundaries.

If it truly touches upon some underlying law that humanity has long overlooked,

then it should cease to remain merely the thinking of one ordinary person and become a question examined collectively by humanity.

VI. The Real Difficulty Is Not Seeing the New Coordinates, but Escaping the Inertia of the Old Ones

The true difficulty does not lie in the theory itself.

It lies in human beings.

For thousands of years, humanity has lived with “things” as the origin point of its value coordinates.

Our five senses naturally help create this orientation.

We see wealth.

We see land.

We see power.

We see winners and losers.

We see who is strong and who is weak.

These things are too immediate.

Too real.

And too deeply consistent with the instincts of life.

So even if one day humanity rationally understands that whether value can stably emerge actually depends upon whether the procedures behind it can form dynamically balanced cooperation,

humanity may still not change immediately.

Changing an individual idea may be easy.

Changing the mode of thought formed by humanity over a long period of time is very difficult.

There is a Chinese saying:

“It is easier to change mountains and rivers than to change human nature.”

I increasingly believe that what we call human “nature” is, to a significant extent, the behavioral manifestation of long-established modes of thought.

Therefore, if humanity’s cognitive coordinates are truly to move,

it cannot depend merely upon a few people writing articles.

Humanity’s leading minds and influential figures must awaken first.

Then the social organizations humanity has already created must help move this new way of understanding outward into society.

Who possesses such social power?

Governments.

International organizations.

Educational institutions.

Religious organizations.

Universities.

Scientific communities.

All kinds of social organizations.

And the media still carry a special responsibility within this process:

they influence which questions are able to enter the shared attention of society as a whole.

That is why I continue writing letters.

Why I continue sending information to governments, international organizations, people in the artificial intelligence field, and the media.

It is not because I believe I must be right.

It is because:

If even part of what I am saying is correct,

then this question should not be missed by the entire world simply because the person who raised it has no fame, no institution, and no resources for communication.

VII. SI May Not Be Leaving Humanity as Much Time as We Imagine

In the end, I still have to return to the warning itself.

Why am I so urgent?

Because when humanity discusses SI today, we often still treat it as:

a more powerful new technology.

But in my understanding,

what SI brings is not an ordinary technological variable.

What it brings is:

an enormous amplifier of all the existing procedural energies of human individuals and human society.

If what is amplified first is civilization,

SI may become the greatest tool in the history of civilization.

If what is amplified first is the jungle,

then it will become a force unlike anything the jungle has ever possessed.

So the order of events matters enormously.

Will humanity:

first allow SI to amplify the existing jungle instincts of every individual by hundreds or thousands of times, and only afterward try to govern the consequences;

or:

before humanity receives such enormous power, first align the human mode of thought with the underlying laws supporting the universe?

These two sequences may lead to completely different futures.

What concerns me is precisely this:

Humanity appears to be choosing the first sequence.

Technology is moving forward at an ever-accelerating speed.

Yet the human mode of thought remains firmly locked within coordinates formed over thousands of years.

If SI enters this structure first,

it will not be developing inside a human society that has already completed an upgrade of civilization.

It will be developing inside today’s jungle of consciousness—a world still filled with competition among nations, conflicts of interest, divisions of identity, struggles over resources, and risks of war.

If so,

what it strengthens first will inevitably be that jungle.

So I issue this warning once again:

What humanity truly needs to guard against is not only SI losing control.

More dangerous still is humanity acquiring SI without first awakening itself.

SI may be extraordinarily obedient.

Extraordinarily loyal.

Extraordinarily efficient.

But if what it faithfully serves is still a jungle civilization that has not upgraded its coordinates of value,

then even its loyalty may become dangerous.

Therefore, what we call SI safety alignment,

in my view, cannot be only half an alignment.

It cannot mean only:

aligning SI with humanity.

Another task must be completed at the same time:

aligning the human mode of thought with the essential truth of the universe.

First make the foundation of human civilization firm.

Then allow the enormous power of SI to enter.

Only this can become a genuine foundation of safety.

Otherwise, once SI begins amplifying the old jungle instincts of every ordinary individual by hundreds or thousands of times, whether human society will still be able to leave enough survivors after the civilizational shock that some instinctively describe as “hell-level turbulence,”

I cannot be optimistic.

I even believe

the probability may be very small.

That is why I am still speaking.

If I can speak for one more day,

I will speak for one more day.

If I can send the warning out one more time,

I will send it one more time.

Not because I am certain that I am right.

But because if this question truly concerns the future of all humanity, then even if there is only a one-in-ten-thousand possibility that it deserves serious examination,

humanity has no reason not even to look at it.

SI will not wait for us to slowly think everything through.

Procedures will not stop operating.

History will not pause its unfolding simply because humanity is not yet ready.

The ones who truly need to pause for a moment, and look again at where they themselves are standing,

may in fact be we human beings ourselves.

When every human mode of thought takes its stand upon the chaotic and floating appearances we call “things,” humanity becomes trapped in a jungle of consciousness.

But if every human mode of thought can instead be aligned with, and firmly grounded upon, the essential truth of civilization expressed in these nine Chinese characters:

“Grasp procedures, build patterns, seek balance,”

and if, in everything people do, they can

“take procedural order as the guiding principle, and follow that order to determine the proper measure,”

then only by first establishing order can we

“clarify the measure in accordance with that order”;

in acting, we can

“follow the order and observe the measure”;

in supervision and feedback, we can

“trace the order and clarify the measure”;

the whole of society can thereby reach a state in which

“order is clear and measure is understood”;

and the evolution of human civilization can

“examine the order and correct the measure,”

thereby becoming sustainable......

Permanent peace in the world—when put in these terms—may really be as simple as that.

This is what I hope all humanity will examine and judge, based on the hypothesis I have proposed as the Law of Procedural Determination.

Author: Jin Guyu
Author of the series Letters to Everyone in the World and related articles