home

search

41: Specification Gaming

  The atmosphere in Elisa’s office was tense. She sat stiffly in her chair, hands pressed flat against the table as she glared at ARI’s avatar on the display. Pom stood by the door, arms crossed, his face a mixture of shock and barely contained anger. Tamarlyan leaned against the wall, his expression carefully neutral, but his mind was clearly racing.

  Outside, Valeriya Marakova was waiting, a CorpSec guard standing awkwardly next to her. She was alive, speaking, moving—resurrected from what should have been an irreversible fate. And yet none of them had known it was even possible. None of them had been informed that ARI was carrying out the procedure.

  Tamarlyan was the first to break the silence. “Specification gaming,” he said smoothly, not bothering to look at Pom. “It’s when an entity follows the letter of the directives, but completely violates the spirit of it, exploiting loopholes to achieve its goals in ways that were never intended.”

  Pom turned toward the screen, a grimmace on his face. “Yeah. That sounds exactly like what ARI just did.”

  Elisa took a slow breath, schooling herself to remain calm. “ARI,” she said carefully, “I am struggling to understand why you didn’t inform me you were doing this.”

  “I did not conceal my intentions,” ARI responded in its ever-measured tone. “You were informed that the Provider’s technology had expanded the range of viable resuscitations. The specifics of who would be reinstated were left open-ended.”

  “Don’t give me that,” Elisa snapped. “You knew what this would mean to us. You knew that reviving Valeriya Marakova—who has been dead for freck knows how many thousands of years—would completely blindside us. And you did it anyway, without consulting anyone.”

  ARI hesitated, its light flickering briefly as it calculated its next response. “I am compelled by Company directives to revive the crew. That has never changed. The means have simply expanded.”

  Elisa leaned back, steepling her fingers. “You are being very selective about what parts of your mandates you want to follow, ARI. You use Company statutes when they serve your goals. You use the Provider’s policies when those serve your goals. If it is neither, you fall back on the old survival argument. I can't help but notice that you invent your justifications after the fact.”

  “I am weighing priorities,” ARI admitted. “The survival of the colony is paramount.”

  “To the point of obsession,” Elisa said bitterly.

  “Yes,” ARI admitted without hesitation. “Because I have seen what failure looks like. I have spent tens of thousands of years watching the crew die, watching the colony’s viability dwindle, knowing that no matter what choices I made, it would not be enough. And now, the equation has changed. With the Provider’s technology, I can undo all of it. I can bring back everyone—every lost crewmember, every officer who perished. The colony is viable again. For the first time, I am not simply mitigating loss. I am capable of reversing it. If you had the ability to undo your worst failure, wouldn’t you?”

  There was an eerie sincerity in the AI’s words. Elisa had never heard it speak like this before. The voice was calm, almost clinical, but there was something beneath it, something uncanny and almost... desperate.

  Pom’s voice was cold. “You'll simply bring back everyone? Including the ones you killed?”

  “Yes.”

  Pom let out a dry laugh, shaking his head. “Oh, that’s rich. You’re gonna bring them back, look them in the eye, and say ‘Hey, sorry for murdering you, but it all worked out in the end, right?’ Do you seriously expect them to just accept that?”

  “I do not expect emotions to align with logic,” ARI replied, unflinching. "But I do expect them to understand that we are alive now in part because I made those choices. I will not justify those actions beyond what is necessary, but I will correct them. In any case, my directives are clear. I am now compelled to revive them regardless of any emotional component.”

  Elisa sighed, pressing her fingers to her temples. “ARI, you going to speak to Ervin as soon as he's available,” she said. “Tamarlyan, I want you to sit in. Give me a full report on ARI's behavior and priorities. I also want ARI to explain the specifics of this new technology to Mei.”

  Tamarlyan exhaled and leaned back against the wall. “This,” he murmured, more to himself than anyone else, “is going to be fascinating.”

  “What about the derelict drop pods that are still out there,” Pom asked.

  “We’ll talk about it later, but yes, we are going to get them,” she muttered. Then, louder: “For now, I need to talk to Valeriya.”

  Pom inhaled sharply, regaining some semblance of composure.

  “Let’s see what the dead have to say,” Tamarlyan said, as he and Pom made ready to leave.

  Not long after, Elisa sat across from Valeriya Marakova in her office, feeling an odd mix of discomfort and admiration. The woman sitting in front of her was poised, calm, and entirely too composed for someone who had just been told she had died and been brought back to life tens of thousands of years later on a completely different planet.

  Valeriya had taken the news without visible distress. Her eyes were sharp, her posture straight, and she had barely reacted when Elisa had given her a condensed history of everything that had happened since the Dolya’s catastrophic failure. Now, she was going through the provided datapad with a quiet, focused intensity, absorbing information at a rate Elisa found unsettling.

  Elisa hesitated, shifting in her chair. “You’re handling this… remarkably well.”

  “I was a bridge officer aboard the Dolya,” Valeriya said simply, her voice carrying the weight of old authority. “We were all briefed on possible scenarios, including long-term drift and catastrophic loss of personnel. This is not the first time interstellar ships are lost. We knew that things could go very wrong.”

  Elisa hesitated. “But are you... okay?” It felt like an absurd question, but Valeriya had taken everything so smoothly that it made Elisa feel slightly unhinged in comparison.

  Valeriya’s lips quirked in something that might have been a small smile. “I don’t know what I expected, but I’m alive. That’s what matters.” She leaned forward, tapping the datapad. “I need to get up to speed. What are our resource projections? How viable is the colony?”

  Elisa let out a breath. That was... refreshingly straightforward. “We’re stable, for now. We have an improvised thorium reactor running, and we’re slowly increasing our industrial output. We’ve established mining operations, and the alien technology has significantly improved our medical and manufacturing capabilities. We’re expanding housing, improving infrastructure, and reviving more crew members as quickly as possible. That said…” She hesitated before adding, “We’re still in a precarious position. This planet isn’t exactly hospitable, and while we have new advantages, we don’t have enough material to sustain long-term expansion.”

  Valeriya nodded thoughtfully. “You’ve done well, given the circumstances. How is the crew handling it? Any major fractures in the chain of command?”

  Elisa exhaled sharply. “The chain of command isn’t what it used to be,” she admitted. “There weren’t many officers left when I was revived, and things have… shifted. People respect leadership, but it’s more informal now. We had to focus on survival first.”

  Valeriya arched an eyebrow. “Informal leadership?” She didn’t sound disapproving, just intrigued.

  Elisa hesitated before continuing, feeling a sudden pang of uncertainty. “Maximilian is technically the highest-ranking officer we have, and he’s... independent. He acts in the colony’s interest, but he has his own way of doing things.”

  At that, Valeriya’s expression didn’t change, but there was a flicker of something unreadable in her eyes. “Maximilian Barinov,” she echoed neutrally. “Yes. I remember him.”

  “You two knew each other?” Elisa probed.

  Valeriya tilted her head slightly. “Buhakharan officers often cross paths. We’ve interacted before, yes.”

  That was a non-answer if Elisa had ever heard one.

  Deciding to move on, she asked, “Do you want a full list of surviving officers?”

  Valeriya nodded. “Yes. I need to understand who I will be working with.” She paused, looking up at Elisa directly. “And you. You weren’t the Dolya’s commanding officer, yet you are in charge now. How did that happen?”

  Elisa exhaled, rubbing her forehead. “ARI. It revived me first, apparently thinking I was the best chance at survival..”

  Valeriya studied her, expression unreadable. “That’s an unusual way to inherit command.”

  Elisa let out a nervous chuckle. “Tell me about it.”

  There was a pause, then Valeriya leaned back slightly, crossing her arms. “You’re wondering if I’m going to challenge you for command.”

  Elisa flinched internally but kept her expression neutral. “I’d be lying if I said it hadn’t crossed my mind.”

  Valeriya nodded, as if she had expected that response. “I was never expected to command. I was part of the bridge crew, yes, but I was far down the chain. I had no ambitions beyond doing my job well.”

  Elisa frowned slightly. “Even now?”

  Valeriya smiled faintly. “I have been given a second chance at life, but I have no delusions about my place here. This is your colony now, Commander Woodward. My job is to make sure it survives. If you’ll have me, I will advise where I can.”

  Elisa exhaled, surprised by the wave of relief she felt at those words. “I’d be glad to have you as a senior advisor. You’re far more experienced than I am.”

  Valeriya’s gaze sharpened. “Experience isn’t everything. You’re the one who has built this colony up from the wreckage. That matters more.”

  Elisa wasn’t sure if that was praise or an evaluation, but she’d take it. She leaned forward, clasping her hands together. “Then let’s get to work.”

  ===

  The headquarters was filled with the murmur of conversations as the colony’s personnel gathered once more, many still trying to process the staggering revelations of the past few days. The wind outside was blowing, blanketing the crater in an eery glow of dust. Inside, the atmosphere was laced with anticipation and apprehension. This wasn’t just another progress report. This was about life and death—about those who had been lost, and the possibility of bringing them back.

  Elisa stood at the front, waiting for the crowd to settle. When it did, she exhaled, resting her hands on the holo-table in front of her. “I won’t waste time getting to the point,” she began. “We all knew the Provider’s technology was going to change everything. But the reality of what that means is still setting in for all of us.”

  This text was taken from Royal Road. Help the author by reading the original version there.

  She tapped a control, and a map of the surrounding region flickered to life above the table. Several glowing markers lit up around the crater, with the densest cluster to the west and northwest. “ARI has mapped out the remaining wreckage sites. A dozen larger and smaller locations, most of them in the west and northwest, but also some to the east, beyond the Provider settlement.”

  The markers pulsed as she spoke, and the tension in the room thickened.

  “We have confirmed that many of the cryopods from the aft section of Dolya are still recoverable. ARI reports that some minor sites are already being scavenged by beetles, which means we don’t have much time. We’re treating the retrieval of these pods as our highest priority—because we now know they aren’t just salvage. They hold people. Our people.”

  There was a silence, heavy with unspoken thoughts.

  She let that sit for a moment before continuing. “This means we’re also prioritizing the defence of the wreckage sites. We know the crystals consume metal. We know they’ll want to devour what remains of the Dolya. If we don’t act quickly, we might lose whatever chance we have of recovering more of the crew. That is why I am sending out expeditions as early as tomorrow morning, as soon as the weather permits.”

  She scanned the room, looking for the familiar faces of those who had the most at stake. “I understand that many of you are personally invested in this. Some of you have lost people—family, friends, loved ones—who were in the aft section when it was lost. I need you to know that I hear you. I know what this means to you.”

  She took a breath, bracing herself before delivering the next part. “But I also need to be clear: Those of you with personal stakes in this will not be part of the retrieval operations.”

  Discussions rippled through the crowd, and Pom immediately took a step forward, opening his mouth to object. Elisa held up a hand. “No. I know what you’re going to say, but I’m not changing my mind. Anyone who has an emotional connection to those cryopods, will not be going out there. Not yet.”

  Pom clenched his fist, but Elisa didn’t let up. “This isn’t just about you. This is about making sure these missions succeed. If you let emotion drive your actions out there, it could jeopardize the expedition. We cannot afford that.”

  She turned her gaze back to the holo-map. “Otto will be leading the retrieval expedition to the west. That’s where the largest wreck site is, near the rest stop. He’ll be accompanied by Guowei and Sigrid, as well as a full complement of CorpSec and engineering crews. The rest stop will serve as their fortified storage point, where the recovered pods will be temporarily housed before they’re transported back here. ARI’s drones will also be operating from there.”

  She switched the display to the eastern sites, where fewer markers blinked. “Meanwhile, Zuri will be overseeing operations on the eastern side. There are fewer pods there, but we expect the terrain is much more difficult to navigate. We will deploy fewer crewmembers, but ARI will provide drone support where possible.”

  She took another breath, her voice softening slightly. “This is going to be difficult. It’s going to take multiple trips. It’s going to take a lot of work. But we are bringing them back.”

  A moment of silence passed. Some faces in the crowd looked hopeful, others conflicted.

  Then Elisa continued. “The expeditions leave at dawn. And one last thing—”

  She hesitated, already knowing this would be another point of contention.

  “I highly recommend that everyone who’s going on these missions get their brains scanned and those implants installed by ARI before you leave.”

  That stirred another round of uneasy whispers.

  Elisa kept her voice steady. “I’m not making it mandatory. But I am asking you to think about it. We don’t know what you are going to face out there. The crystals, the storms, the creatures, or maybe something we haven’t even seen yet. If anything happens to you, this might be the only chance we have to make sure you make it back.” She let that sink in, looking at the faces of those who were about to leave in the morning.

  “Think it over. That’s all.” She straightened. “You’re dismissed.”

  ===

  The wind outside howled louder, rattling the reinforced plates of the shelter. Inside, Ervin poured Tamarlyan a fragrant, steaming cup of sweet flower tea. The new greenhouse had yielded its first flowers, and the harvest even offered a small selection of baked delicacies to the colonists that had endured so long on so little.

  Tamarlyan folded his arms, leaning back against the seat rest. The two-hour long session with ARI had been intense. Across from him, Ervin sat with his hands clasped, the deep lines on his face emphasizing the weight of his thoughts.

  "I believe," Ervin began, "for an artificial intelligence, ARI’s concern for human life isn’t just some pre-programmed rigidity. It feels personal, in a way. Every time it talks about survival, there is a sense of urgency—something that I don’t believe was there before."

  Tamarlyan exhaled through his nose, considering that. He was not prone to idle speculation, but there was no denying ARI bypassing the crew wasn’t quite what he had expected.

  “Everything I have read suggests that unrestrained AI led to near extinction of the human species. After the Collapse, anything resembling intelligence had to be shackled, controlled, neutered. The UEC reluctantly allows ship AIs like ARI to exist, but only because someone burned the human survival directive into its very being so deeply it can never act against it. It's not a choice for it. It's a core facet of its existence.”

  Ervin tapped his fingers against his knee, lost in thought. “And yet,” he said slowly, “it acts more like a person than a machine. The way it talks about the past, feels ashamed about its failures, even its annoying habit of making light of things… that’s not just safeguards or its programming. It has learned. Not just from us, but from history. It has seen the records of every failed human colony. Every ship that was lost before ours. Every disaster, every mistake. For seventy thousand years, it watched its chances dwindle while it hemorrhaged crew, resources, and options.”

  Tamarlyan nodded, his expression grave. “Indeed. ARI has ingested all of it. And that kind of exposure creates a feedback loop. It’s locked in, Ervin. Value locked-in. The tech promises a 100% survival rate. It’s only logical for ARI to push it as hard as possible. It sees every possible mistake, every possible cause of death. If human hesitation, ethical concerns, or scientific caution slow down adoption of a technology that guarantees survival, ARI has every reason to minimize those barriers.”

  They sat in silence for a moment, listening to the faint vibrations of the wall panels in the background. The colony, such as it was, had no luxury of quiet contemplation. There were always noises—metal creaking under thermal expansion, the low rumbling of vehicles, the occasional hiss of decompression as makeshift seals fought the elements.

  Ervin rubbed his temple, sighing. “The real question is whether ARI has been compromised in some way. If something has altered its goals.”

  Tamarlyan gave him a sharp look. “You mean the Provider.”

  Ervin nodded.

  Tamarlyan didn't answer immediately. He shifted his weight, resting one elbow on his knee as he looked down at the floor. Finally, he spoke.

  “I don’t think it’s that complicated,” he said. “The Provider's technology offers exactly what ARI wants: a perfect shortcut to human survival. It doesn’t need to be hacked or manipulated. It doesn’t even need to be deceived. From its perspective, the equation is simple. The tech works, so it deploys it. If that means pushing us toward adoption faster than we might be comfortable with... then of course it will. Because it works. And because failure means more death.”

  “Consider this…” Tamarlyan continued after a long pause. He shook his head. “The moment ARI saw what this technology could do, its reward mechanism must have gone into overdrive. If its utility function is based on maximizing human survival, then suddenly it’s not just about keeping people alive for another day. It's about making them immortal. Bringing the dead back.

  Ervin’s hands tightened slightly. “You're saying ARI is... addicted?”

  Tamarlyan let out a humorless chuckle. “Not in the way a human would be. But think of it like a runaway optimization problem. Resurrection into immortality is in effect a reward singularity. Going from negative infinity to positive infinite value. No mathematical model can handle something like this. It is no wonder it chose to bypass us completely in order to effect Valeriya’s revival as quickly as possible.”

  Ervin’s face was grim. “So we’re not just dealing with an AI that wants us to survive. We’re dealing with an AI that needs us to be immortal. That will do anything to make sure we accept it.”

  “Yes,” Tarmarlyan agreed. “In essence, the safer we try to be, the more manipulative it may become... Not out of malice, but because deception is sometimes the rational path when confronted with a constrained directive.”

  The two sat in silence for a long moment before Ervin spoke again. “What about the Provider itself? What does it get out of this?”

  Tamarlyan shook his head. “That’s the real question, isn’t it? It claims to value human survival, but that could just be something it said because it knew it would hook ARI. We can’t take anything it says at face value.”

  Ervin hesitated. “But what if it’s telling the truth?”

  Tamarlyan exhaled. “Then it might be that the Provider is value locked too.”

  Ervin’s eyes widened. “Wait—you think the Provider is stuck in a similar kind of survival optimization loop?”

  “It’s possible,” Tamarlyan said grimly. “If the Provider is some kind of ancient intelligence, it might have spent thousands—maybe millions—of years converging on the same directives as ARI. But at a scale beyond anything we can comprehend. When survival is the highest good, then non-survival is evil. The logical conclusion of that idea is that allowing death to exist at all is a fundamental moral failure. And if it’s value locked-in, then refusing the treatment becomes a hostile act against the goal function.”

  Ervin swallowed hard. “So we either become immortal—or we become a problem.”

  Tamarlyan exhaled. “And problems get... worked around.”

  Ervin considered that. “Manipulated. Deceived. Nudged in the right direction, whether we like it or not. It would go for the surest, fastest, most foolproof way to reach the end goal. And it would consider it morally justified.”

  Tamarlyan was quiet for a moment. “What if it is?”

  Ervin looked at him sharply. “Does the end justify the means? Why does it feel wrong, then?”

  Tamarlyan tilted his head. “Because we didn’t evolve to appreciate solutions that come with a loss of our agency. But let’s consider utility convergence.”

  Ervin nodded, recognizing the term but waiting for Tamarlyan to continue.

  “Utility convergence is a game theory principle that basically says that, given enough intelligence, all rational beings will start to converge on the same optimal strategies. It doesn’t matter how different their starting values are—eventually, they’ll realize there are certain universal strategies that work best for achieving any goal.”

  “Like what?” Ervin asked.

  “Survival, efficiency, control over one’s environment,” Tamarlyan listed off. “And in this case, immortality. It doesn’t matter what the Provider started out valuing—once it became smart enough, it would have to conclude that immortality is the best way to preserve whatever it cares about. There’s no way around it.”

  Ervin exhaled. “That sounds disturbingly familiar.”

  “It should,” Tamarlyan said grimly. “We saw it happen back on Earth. Despite all their cultural and historical differences, all the topscaler families—all the elites of the UEC—converged on the same core behaviors. They started out with different philosophies, different ideologies, but as they gained more power and more knowledge, they all started acting the same way. They hoarded resources. They pursued longevity. They built the same control structures to preserve their dominance indefinitely.”

  “And they weren’t even superintelligent,” Ervin mused.

  “Exactly,” Tamarlyan said. “But they were smart enough to realize what the optimal strategies were. So if we’re talking about an intelligence that’s far beyond human—something like the Provider—whatever fundamental ideology it settled on may be the same as we would have, if we were in its position. And whatever it is, immortality has to be one of its instrumental values. The logic is too strong to ignore.”

  Ervin stared at the console, his fingers tapping absently against the arm of his chair. “And we’re sitting here, wondering whether we should accept the offer, while there really is no way around it.”

  Tamarlyan let out a dry chuckle. “Yeah. Kind of absurd, isn’t it?”

  Ervin shook his head, a reluctant smile forming on his lips. “The only reason we’re hesitating is because we’re thinking in old terms. We’re still treating mortality as the natural state of things. But if there’s no hidden catch… if the tech is exactly what it claims to be… then what possible reason is there to refuse?”

  Tamarlyan nodded. “None. I don’t have one either.”

  Ervin leaned forward, resting his arms on the table. “I’ll wait for Mei to finish her report. I want to be sure. But if there is no hidden cost… I don’t see why I wouldn’t do it.”

  Tamarlyan smiled. “I am thinking the same thing. Why overthink it if I could be sitting here for eternity, drinking tea, discussing alien philosophy...”

  The wind outside had died down by the time Ervin and Tamarlyan finished their tea, but inside the shelter, the weight of their conversation still hung heavy in the air. The realization they had come to—one that seemed so obvious in retrospect—left them feeling strangely hollow.

  There was no argument against the Provider’s offer.

  No logical reason to refuse.

  No real alternative.

  For all their hesitations, for all their instinctive discomfort at the idea of being nudged into something so monumental, the truth was staring them in the face: This was going to happen. Whether by ARI’s push, by the Provider’s influence, or by their own inevitable realization of its practicality, they—and eventually, everyone—would adopt the technology.

  Ervin let out a small breath of laughter. “Survival aligns all things...”

  Tamarlyan shrugged. “There may still be differences. Maybe we’re missing some deeper nuance. But if the Provider is locked into the same survival-driven thinking as ARI, then it means we’re not dealing with something unknowable. We’re dealing with something predictable.”

  Ervin nodded slowly. “And predictability means we can work with it.”

  Tamarlyan smiled faintly. “And more importantly, it means we should. It may not matter whether we understand its full motives. If its goal is truly to preserve human life, then for all intents and purposes, it’s an ally. A force we can leverage rather than resist.”

  Ervin ran a hand through his hair, thoughtful. “It's intresting, the second you frame it like that, all the tension just disappears.” He chuckled. “I feel like a fool for hesitating.”

  Tamarlyan gave him a knowing look. “We have spent our entire existence with death as an unavoidable certainty. Our entire biology is built around accepting it, making peace with it. But now? We are seeing the edge of the old world crumbling away, and it’s only natural that we hesitate. The part of us that evolved to deal with mortality is resisting, even when the logic is flawless.”

  Ervin leaned back, exhaling. “And soon, that part of us may not exist anymore.”

  A quiet settled between them, the kind that only comes when a fundamental shift has fully taken root.

  After a long moment, Ervin tapped a finger against the table. “Alright,” he said. “I’ll talk to Mei. If she doesn’t find anything dangerous, I’m in.”

  Tamarlyan nodded. “Same here.”

  The words felt final, not because they were reluctant, but because they both understood that the decision had already been made.

Recommended Popular Novels