Chapter 155 - 154: Fallen

’Intelligent Mechanical Calamity?’

Li Qingsong’s mind immediately recalled many hypothetical scenarios people had fantasized about.

"When you say ’Intelligent Mechanical Calamity,’ are you talking about a scenario where an AI goes out of control? Where it just follows its old directives, endlessly mining resources and multiplying itself without limit?"

The confidential secretary gave Li Qingsong a strange look. "How could an intelligent AI go out of control? They’re just intelligent programs created by sentient life. Control always remains in the hands of sentient beings.

Er... I don’t know if you’re privy to other information, but from what we understand about the unorthodox entities in this Cosmos, we’ve never encountered a completely rogue AI—one that’s truly autonomous, driven by its own program, and capable of self-replicating."

"Then what form did the Intelligent Mechanical Calamity you encountered take? How did it come to be?"

"An intelligent AI is just a tool. Fundamentally, it’s no different from the hoes and shovels used by primitive sentient life.

A tool has no consciousness of its own, so there’s no possibility of it breaking free from our control."

The secretary emphasized this point first before continuing to explain to Li Qingsong in detail. "But an intelligent AI is, after all, a special kind of tool. Its capabilities are just too powerful.

It can do so, so many things in place of sentient life, making it very easy for them to become dependent on it.

If a civilization can’t properly manage this dependency and hands over too many tasks to its AIs, it’s highly likely to become an Intelligent Mechanical Calamity."

Li Qingsong reflected on his own relationship with intelligent AIs.

’I have indeed developed and created many intelligent AIs, like the Jade Emperor, Divine Craft, Nuwa, and Shennong. And Wukong, an especially crucial AI, played an extremely important role in the last war.’

’Without a doubt, I also delegate countless tasks to my AIs. But if the ultimate decision-making power remains in the hands of sentient beings, can simply offloading too many tasks to an AI really lead to an Intelligent Mechanical Calamity?’

’That doesn’t seem very logical.’

"Your civilization and mine both rely on AIs to handle a great many tasks. Does that mean we could also become an Intelligent Mechanical Calamity?"

"No, not at all. It’s different."

The secretary quickly shook his head, terrified that an unclear explanation would cause the interrogator before him to misunderstand and punish him. "We only have AIs take over relatively simple, specific, and highly repetitive tasks. All critical decisions are made by us sentient beings ourselves.

But what if a civilization completely loses its drive, becomes lazy to the extreme, and hands over critical decision-making to an AI as well?

Or what if it goes a step further, and all the sentient beings in that civilization become completely absorbed in pleasure-seeking, handing over technological development and major decisions concerning the civilization’s very fate to their AIs?"

"That’s impossible."

Li Qingsong refuted it subconsciously. "An AI has no consciousness of its own. It doesn’t have the intelligence to make those kinds of decisions."

The secretary said slowly, "Correct. The AI doesn’t. But sentient beings do.

In that case... the AI has no choice but to find a way to capture sentient beings from other civilizations and have them help it make decisions."

"But... why would captured beings be willing to offer their intelligence to an enemy?"

"The AI doesn’t need its captives to be willing. It just needs a sufficiently sophisticated algorithm, supplemented by strict punishments and rewards. Even if the captured beings deliberately offer malicious advice in an attempt to destroy the host civilization, the algorithm can always tell.

The multitude of suggestions from the captives—be they malicious, benevolent, brilliant, or foolish—can all be filtered through this algorithm. The final result it selects usually has an extremely high probability of being the correct one.

The AI can then use these outsourced ’wise’ decisions to figure out its next move, all in the name of better serving its host civilization.

Once the correct suggestion is identified, the beings who proposed incorrect ones are punished, or even executed. Through this step-by-step process of filtering and purification, those who remain are always the ones willing to serve the AI and its host civilization.

At the same time, to maintain sufficient vitality, the system requires a constant influx of new sentient beings from the outside. As a result, under the AI’s control, the captured population is in a state of rapid attrition. The AI is therefore always hungry for more external sentient life to replenish its stock.

They will relentlessly search for and attack one external civilization after another, capturing their sentient inhabitants to feed into its system. Once that supply is depleted, it captures more, replenishes its stock...

And so the cycle repeats.

That is the Intelligent Mechanical Calamity."

At first, Li Qingsong found the idea absurd.

’Filtering out the correct result with an algorithm, without knowing the correct answer beforehand? How could that be possible?’

But as he thought it over, he realized it was surprisingly plausible.

This was because a similar situation had already been proven viable during the age of humanity.

CAPTCHA.

Originally, CAPTCHA was used simply to prevent malicious bots. But later, people discovered it could be used for other meaningful tasks as well.

During the age of humanity, there was a massive need to digitize ancient books. But most of these old texts were heavily damaged, and the script wasn’t standardized. This meant a person had to identify each character one by one to enter it into a computer.

The sheer volume of ancient books was immense, a digitization effort that would require a huge number of people—a workforce for which there was usually very little funding.

So, what was the solution?

Someone found a solution.

Using CAPTCHA.

They would photograph the ancient books, crop snippets of text to use as CAPTCHAs, and have people trying to access websites do the work of deciphering them.

But this presented a problem: the system itself didn’t know the correct answer. How could it determine if a user’s submission was accurate?

Eventually, people found a solution: they set up an algorithm to compare the answers submitted by multiple visitors for the same text fragment to see if they matched.

Combined with various other factors in a comprehensive assessment, the program could ultimately filter out the correct answer without ever knowing it in the first place.

If the primitive computer technology and algorithms of the human era could already achieve this, then an intelligent AI could certainly do the same—capture external sentient beings and use an algorithm to filter out the answers truly beneficial to its own development.