AI Controls Support AI Demand Surge

The rate of technological innovation is rapidly increasing. What was revolutionary a decade ago is already table stacks or even outdated today. The next decade will be no exception, especially with advancing artificial intelligence technology.

Large language models (LLMs) get a lot of attention now, primarily because of their user interactivity. But artificial intelligence applications that safely automate high-value, repetitive and complex tasks are quickly and somewhat quietly gaining traction in the industrial sector. They may ultimately prove to be more valuable in the long run.

One significant application relates directly to precise and efficient temperature control for industries requiring cooling that is critical to the business’s operations. Also known as “mission critical cooling,” this particular application of precise control poses a large potential value for industries like pharmaceutical production, district energy and data centers.

Within the data center industry specifically, recent demand growth has led to an arms race for more land and power for new data center construction as well as an increase in server density for existing data centers. With this increase in IT density, heat production inside data center facilities is becoming a more difficult and dynamic challenge. Rapid and proactive management to handle heat loads in the data halls is crucial to uptime and equipment safety. As these existing facilities shift and run closer to full capacity, the need for efficient and reliable cooling management becomes paramount.

Artificial intelligence (AI) is not only driving this demand uptick but is quickly becoming the revolutionary force that can enhance how data centers operate.

To provide insight into how Phaidra sees AI-enhancing data center operations, here are a series of possible journal entries by a fictional data center operator. These entries are fabricated but echo responses and conversations our team has had with industry insiders.

Log Entry: A Skeptical Beginning

When I first heard about integrating AI control for our cooling system, I thought it was just another trend. I’ve seen plenty of tech trends come and go—many of them do not live up to the hype. And the promise of AI “revolutionizing” our operations sounded ridiculous and potentially threatening.

I remember sitting in the meeting room, listening to the presentation on AI. The enthusiasm was palpable, but so were the doubts.

Could AI really handle a complex, high-stakes environment like a mission-critical data center? Reliability was crucial, not some fleeting fad. Would it go rogue and run things in a dangerous way, threatening stability and availability? I’ve seen enough movies to be cautious of “AI control.”

Besides, what about myself and my colleagues—the existing operations team? Would this replace us if it actually managed to meet the promise it was making?

So many questions circled in my mind.

The idea of an AI control system, or as they’re calling it, a “Virtual Plant Operator,” feels like a leap into the abyss. I spent years learning the ins and outs of our operations. There are things we do here that are not even fully documented because we know what needs to happen and when. Now I’m supposed to trust a machine to make decisions that could impact or even threaten everything. How will we know this is actually reliable?

It was a lot to take in. But I understood the potential.

If AI control could handle the repetitive, mundane tasks, it would free us up to focus on the bigger picture. We could spend less time putting out fires and more time strategizing.

Still, the skepticism lingered.

Would AI deliver on its promises? Or would it be another overhyped tech trend?

With cautious optimism, and a little insistence from management, our operations team decided to give it a chance.

Here’s to hoping that AI lives up to its promise and transforms our data centers for the better.

Log Entry: Technical Debt Cleanup and Safety Constraints

In order for an AI control system to operate safely and efficiently, it must have good historical telemetry data from our operations to “train” on and, of course, good live data to make decisions based on. Without good data, it cannot provide good outputs.

I really thought we were doing just fine with all the data we were already collecting in our BMS and data historian. I was wrong on that. Let’s just say we have a solid amount of technical debt built up over the last few years. Items like sensors not being calibrated properly, trends not semantically tagged, a maintenance system not integrated with the BMS, and even just gaps in sensor coverage. We needed new meters at the individual chiller level. I didn’t realize we were only tracking power universally!

Would have been great to run across this AI Readiness article a year or so back during our last re-commissioning.

We’re getting things cleaned up now. After this AI control onboarding process, we should have a very clean and valuable data set to draw from moving forward.

I also appreciate the clarity on exactly how the AI will operate things safely. Every data center operations team knows that the programmed logic in the LCS maintains a specific set of limits or boundaries to keep things stable. These boundaries are referred to as ‘constraints’ —essentially the rules the AI must follow.

In order for us to build some trust with the AI, we will first overconstrain it so that it works well within the bounds of our system. For example, the setpoint limits for the first few months of autonomous AI control will be well within the actual customer SLA limits we have with our hyperscale tenant here. That certainly helps me sleep a bit better at night.

Log Entry: Up and Running

We’ve been up and running in autonomous AI mode for a few weeks now, and it’s incredible what we’re seeing.

This industrial AI control builds on the foundation of our existing PLC system without replacing the existing hardware.

With AI, it’s no longer just about setting up a sequence of operations (SOO) and letting it run for a few years before testing, commissioning, and updating it to regain temporary optimization.

As real-world changes and equipment degradation occur in the facility, our current SOO will become outdated and less efficient. This AI system responds dynamically, making real-time adjustments based on changing conditions and creating perpetual optimization on top of the SOO programmed into our controls. This means the AI control can make subtle changes to setpoints at the right time, every time, helping us maintain stability and reduce downtime risk.

Our SOO doesn’t go away, though. It acts as a limiter that validates any of the AI control setpoint changes as a final safety check to make sure no outrageous changes are coming from the AI. In the case of any connection interruption or if one of our operators makes a manual change that is outside the constraints we’ve assigned the AI, it simply turns itself off with a bumpless transfer back to the local SOO until we’re ready for it to be on again.

We’ve also noticed the AI has begun testing the edges of its constraints a bit, as this is typically where the most efficient operation strategies are found. We have overconstrained it quite a bit so it can better understand all the nuances in our system. But I know we intend to loosen those constraints after a few months of stable control.

I’m excited to see what new strategies this thing may figure out.

It reminds me of when we moved from a manual thermostat to a smart thermostat at home. After a few months of learning our family’s daily routine and understanding the changes outside, it began making simple adjustments—doing something a bit earlier or later than I manually would to keep our comfort at a premium. Plus, it also saved us a few bucks on the energy bill.

There was a moment where a technician manually turned on a pump, and the AI adjusted another valve to avoid a pressure instability event. We actually didn’t even think about that downstream impact right away. The AI caught it 20 or so minutes before we would have!

Reminded me of the cruise control in my car that reduces speed when it senses a vehicle in front getting too close. This self-shift maintains safe operation.

Log Entry: Some fear, then acceptance, then excitement

AI control is certainly a leap in functionality and efficiency. I no longer need to or am expected to make constant manual adjustments. Instead, I am simply overseeing a system that optimizes itself, allowing me to focus on more strategic tasks.

One of the most fascinating aspects of integrating this has been the human factor.

Some of our operators fear the AI will take away their jobs. They worry about losing control and having the skills and nuanced knowledge they've honed over the years become less valuable.

I get it—telling someone to take their hands off the steering wheel and trust the AI can be terrifying.

But on the other hand, there are operators like myself who are now excited about the potential this AI control is showing. I don’t want to sit and stare at screens all day, doing repetitive tasks and trying to stay ahead of any potential bumps in stability. My virtual “colleague” can handle that now.

Even as we’ve relaxed the constraints on the AI control to more closely resemble our customer SLAs, the service has yet to cause any stability issues and has found some unique ways to boost our energy efficiency. Our C-suite folks are always excited about a PUE decrease, and we’ve been able to report at least a 0.05 drop in the first 6 months!

Other operators, like myself, see AI as a way to make their jobs more interesting and impactful. We’re eager to manage, mentor, and coach these virtual plant operators—much like guiding a smart and attentive new hire.

The key is finding a balance.

AI can handle the mundane tasks that put the eager mind to sleep. It frees us to focus on strategy and big-picture outcomes. This shift not only makes the work more engaging. It also ensures that we, as humans, are still in the loop for critical decision-making. We have more time available to us to handle all the smart-hand maintenance our facility requires. I’m actually ahead of my task list now for the first time in a long time.

It’s about enhancing human capabilities, not replacing them.

Log Entry: From Operators to AI Mentors

Picture this: in the data centers of the future, human operators won’t just be managing machines—they’ll manage AI systems.

Imagine being less of a traditional operator and more of a mentor to these virtual plant operators, guiding them to perform at their best.

In these advanced facilities, every human operator becomes an AI control manager.

People might not be managing other people, but they’ll be working with human colleagues across multiple regions to manage these virtual teammates, making decisions and taking actions on their behalf. It’s like mentoring a team of genius assistants.

Like any advanced machine learning tool, learning how to prompt and guide these AI systems is key. This is how you get them to perform optimally on your behalf.

This shift means you won’t have to sweat the small stuff anymore. AI will handle routine, mundane tasks, freeing you up to focus on the big picture.

Strategy, long-term trends, and critical decisions will be where your expertise shines. The AI systems will manage the nitty-gritty details, but myself and the rest of the operations staff now get to be the ones deciding when and how to adjust the rules to fit business needs, safety requirements, and new goals.

I sympathize with those nearing retirement who feel uneasy about this change after investing years into mastering their role. But as we’ve rolled out this service at other facilities with experienced staff nearing retirement, we’re finding that if we had not digitized their operational strategy within this virtual plant operator, we may have lost valuable operations insight as they retired.

There’s a growing need for younger folks who’re hungry for more than staring at screens or doing repetitive tasks. They’ll want something more engaging, and this new AI-driven environment offers just that. Plus, the operational strategies the AI system learns from are in the historical telemetry it ingests. This ensures that these newer operators essentially have the experienced guidance of the most tenured senior operations staff, who’ll be enjoying their well-deserved retirement.

It’s similar to having a Jarvis-like assistant from the Iron Man movies. This smart colleague sits within the control system itself, ready to take action when needed. It provides an extra cognitive capability to consider current and future predicted conditions—either helping you make better decisions or just making them autonomously while you’re off handling maintenance or vendor needs around the data halls.

That’s the kind of relationship we’re building between human operators and AI systems.

Of course, there’s always the fear of making mistakes. People worry about AI making errors and who will be held accountable. It’s a valid concern.

But let’s not forget that humans make mistakes, too.

The idea is to build trust and show that AI can be reliable by having it operate within overconstrained boundaries first.

In the end, it’s about balancing human expertise with advanced AI capabilities. This partnership is making our data centers more efficient, innovative, and resilient. By embracing AI, we’re contributing to our roles by focusing on what we do best. The AI handles the mundane and repetitive with better precision and timing than we ever could.

Much of the recent demand boom in data center capacity is due to the increasing demand for AI applications. It makes perfect sense to leverage AI technology to help data centers respond to this increase in demand dynamically and efficiently.

Log Entry: A Blast from the Past

We've had virtual plant operators deployed at different facilities for over a year now. But somehow, a previous employer came to mind. They operated data centers entirely manually, without any automation. It feels like that must have been decades ago, when it was only a few years.

This company’s approach is to handle everything by hand—chillers, pumps, and system components all require manual intervention.

I couldn’t help but remember the challenges and inefficiencies inherent in that setup.

When they need to start an extra chiller, someone has to manually switch it on. If an issue arises in the middle of the night, it requires immediate human intervention, no matter how minor the problem might be.

This manual process is both time-consuming and prone to human error. It’s usually a cause for anxiety as well. When operators “clock off” for the day, they aren’t actually off. There’s still a sense of potential doom.

It reminded me of a particularly chaotic day when several systems required simultaneous attention. The staff was running around, trying to keep everything in check. It highlighted the stark contrast between manual operations and what I’m seeing now with AI autonomous control.

In a modern, automated data center, AI seamlessly manages these tasks, preventing the frantic rush to fix problems.

The inefficiencies of this manual approach are clear. It’s labor-intensive and costly, not to mention the increased risk of downtime due to human error. Relying solely on manual operations is simply not sustainable. Even more jarring when I remember that the commitment to manual operation was a result of a human error made during a failed automation project.

AI control isn’t just about making things easier. It’s about making them better, more reliable, and more efficient. This blast from the past was a powerful reminder of how overly conservative control paradigms can sometimes be a handicap.

Log Entry: Wide Roll-Out

Well, the first few locations we deployed AI control at have been seeing nothing but positive results with no service interruptions. Better telemetry data is being collected thanks to early implementation processes. Control systems at each location have constraints set to ensure the AI runs safely. Most have relaxed those constraints a bit to let the AI control explore strategies to reduce energy consumption.

We’ve even been able to oversubscribe some locations a bit more now that cooling energy per MW has been reduced. Operators, like myself, at each early deployment site are calmer and able to handle a lot more of our smart-hand maintenance and customer tasks faster.

Other sites that are now beginning appropriate preparations—data cleanup company-wide alone—are going to be extremely valuable long-term. Sites that aren't operational but also about to enter commissioning are now ready for AI control deployment.

But this is where things are getting really fascinating. Training AI agents is nothing short of fascinating.

At existing locations, like mine, we had to first clean a large dataset, impute data points for gaps in our sensor coverage, and get our controls vendor to make some LCS updates to accommodate the AI control needs. But at these greenfield sites that are beginning commissioning, there’s no historical data for the AI to ‘train’ from. What’s fascinating, though, is that during the commissioning process, the cooling system is throttled quite a bit to ensure it'll work properly during operation. So now, instead of training from a year’s worth of consistent control data, these new AI deployments will have only several weeks of data. BUT that data will be enriched as there will be much more equipment stress testing.

This enriched data, I’m being told, will allow the AI to perform better and faster. It’s kind of like an AI chess computer learning from a year’s worth of gameplay. There were 10 million games played with amateurs. Then, one with a few months’ worth of 1 million games played against grandmasters.

With these new builds, we’ve learned to incorporate better sensor coverage and set up our clean data policies correctly from the start. These new virtual plant operators will not need as many constraints and should deliver both stable control and energy savings almost immediately.

I’ve learned a lot about the core aspects of these virtual plant operators that make them so effective at improving over time. At the heart of this AI is a type of machine learning called "reinforcement learning". It’s all about rewarding the AI for making the right decisions and penalizing it for mistakes. Over time, the AI optimizes its actions to achieve the best possible outcomes. And most importantly for us, it always follows the rules it’s provided.

Log Entry: Cognitive Strategy

Integrating AI into data centers is one of the most exciting developments I've seen in my career. One of its greatest benefits is how it lifts the cognitive burden from human operators.

By handling mundane tasks, AI allows us to focus on impactful work. This boosts productivity and makes our role all the more valuable.

Thinking back to the typical day before AI, we used to spend hours monitoring gauges, checking temperatures, and making sure all systems were running smooth. These tasks are repetitive and mentally draining.

They left little room to engage in higher-level strategic thinking or problem-solving. I remember talking to other operators, exhausted from this endless cycle of routine checks.

With AI, this scenario changes dramatically.

The system continuously monitors all parameters, making real-time adjustments as needed. For example, if a cooling system shows signs of inefficiency, the AI detects this early and takes corrective action. This proactive approach prevents minor issues from becoming major problems, ensuring optimal performance 24/7.

This shift has made a huge difference in operational efficiency. Now that we have AI controls in place at nearly all our locations, we also have an incredible, almost company-wide set of highly detailed performance data. So much so that I’ve heard the way we'll design future centers will be much more efficient and faster to operate because we can get closer to right-sizing the cooling system! This can cut months and millions of dollars off the timeline and budget for construction.

With AI handling common cognitive burdens on our staff, we’ve made a significant impact on all sites. There's a potential large positive change in future builds.

Final Entry Thoughts

AI is reshaping mission-critical cooling management. We're seeing changes from manual programming logic updates to dynamic automation that continuously improves. This is just the first application for our industry. Water use, power storage and deployment, and maintenance optimization are all coming.

This collaboration enhances efficiency and drives innovation. It ensures more reliable and sustainable data center operations.

By creating smarter, more adaptive systems, we create a more dynamic and resilient industry.

As we continue to advance, the synergy between human expertise and AI technology is promising. I see a future where data centers are more efficient, progressive, and responsive.

It’s an ongoing path, and I’m optimistic about the incredible possibilities that lie ahead.