Everyone wants to talk about GPUs. How many do we need? Which ones should we buy? Where can we get them? The next generation: how much faster will it be?
I get it… Compute matters enormously to AI. You can’t build serious AI infrastructure without it. But I think all the attention on GPUs is creating a significant blind spot. Once AI leaves the lab and starts doing real work, organizations quickly discover they don’t just have a compute problem. They have a data problem. And it gets worse every time someone uses AI.
AI applications don’t simply consume the data an organization already has. They create more of it. Prompt histories. Inference logs. Vector databases. Generated images, audio, video, and documents. Metadata. Security logs. Compliance records. Backups. All of that data has to go somewhere. It has to be moved, stored, protected, secured, backed up, retained and, when the AI needs it, retrieved quickly.
That’s why I don’t think the AI infrastructure conversation can stop at compute. In many environments, the bigger long-term challenge will be everything around it.
AI creates more data than most organizations expect
Think about what happens when an AI project actually succeeds.
During development, you’re usually dealing with a controlled dataset, a limited number of users and a relatively predictable infrastructure footprint. Everything looks fairly manageable. Then people start using it.
A customer service application goes from a pilot to handling hundreds of thousands or millions of interactions. An internal AI assistant starts indexing constantly changing corporate information. A fraud detection system processes transactions around the clock. A generative video application starts producing enormous files every time somebody clicks a button. Suddenly, every interaction leaves a trail.
There’s the original data. The retrieval data. The prompt. The response. Inference logs. Application telemetry. Security logs. Metadata. Audit records. Different versions. Backups. Some of it might only need to exist for a few hours or days. Other information may need to be retained for years because the business or a regulator demands it.
And here’s the part that I think gets overlooked: the more successful the AI application becomes, the bigger this problem gets.
That’s why AI capacity planning can’t be just about how much compute you’ll need. AI doesn’t just process data. It continuously creates more of it.
Storage isn’t just about capacity anymore
The obvious response to all of this new data is, “Okay, we’re going to need more storage.” True. That’s the easy part. Here’s a better question: What does the application actually need from its data?
Performance matters. AI systems constantly read, write, and retrieve information. Depending on the workload, throughput, IOPS and latency can directly impact application performance.
You can have all the GPU horsepower in the world, but if you can’t feed those GPUs data fast enough, you’ve built a very expensive system that spends part of its day waiting.
Then there’s protection. What happens when the AI system becomes critical to the business? We’re talking about backup, disaster recovery, immutability, ransomware protection, replication and recovery objectives suddenly. Once AI becomes part of day-to-day operations, losing the data around the application can be every bit as disruptive as losing the application itself.
Then ask where that data actually lives. How quickly can the application get to it? Do users or systems in multiple locations need access? Are compute and storage sitting hundreds or thousands of miles apart? Are we moving enormous amounts of data back and forth simply because nobody thought about data locality when the system was designed? This hurts both cost and performance.
And, of course, there’s cost. Not every byte of AI data belongs on the fastest, most expensive storage you can buy. Organizations need to know what’s active, what can move to a lower-cost tier, what has to be retained, and what can eventually be deleted (not just “can” be deleted, but “should” be, for cost, legal, and other reasons…).
At small scale, those decisions don’t seem particularly important. At AI scale, they can be worth a fortune, or cost a fortune. That’s why storage is becoming less of a purchasing decision and more of an architectural one.
Buying more disks solves a capacity problem.
Designing the right data architecture solves the business problem.
The network becomes part of the AI application
How about the network? That’s another piece of AI infrastructure that gets far less attention than GPUs. AI moves data. A lot of it. Data moves between users and applications, applications and models, models and storage, private infrastructure and public clouds, and across regions, data centers, and edge locations. Every time that data moves, the network matters.
How much bandwidth is available? What happens at peak utilization? Where are the users? Where is the application? Where is the data? What happens when a network path goes down? Bandwidth, latency, congestion, fiber diversity, route diversity, and peering… not exactly the glamorous side of AI. But they can make the difference between an application that feels instantaneous and one that feels painfully slow.
With real-time inference, that’s especially true. The person using the application doesn’t care that there’s an incredible GPU cluster sitting behind it. If they ask the AI something and spend too much time staring at a spinning wheel, the experience is bad. Did the delay come from compute, storage, networking, or a poorly designed architecture? They don’t know or care. They just know it’s slow.
And here’s an interesting irony: as AI gets faster, the rest of the infrastructure becomes more noticeable. The faster the model responds, the more networking, storage, and data movement determine what the user actually experiences.
Start planning for the data before it arrives
This is where I think MSPs have a tremendous opportunity.
Customers are already coming to you with AI questions. They’re asking about GPUs, cloud platforms, models, and compute capacity. Keep having those conversations. Just make the conversation bigger.
Don’t only ask, “How many GPUs is this customer going to need?” Ask:
- How much data will this workload generate?
- Where will that data live?
- How quickly will it grow?
- How long does it need to be retained?
- Who needs access to it?
- What security and compliance requirements apply?
- How quickly does the application need to retrieve it?
- How does backup capacity scale as production capacity grows?
- What happens if the primary environment becomes unavailable?
The most important question is perhaps the one customers are least likely to ask themselves: Three years from now, what does this environment look like if the AI project succeeds?
That’s a very different way of thinking about infrastructure. We’ve spent decades designing infrastructure around what happens when things go wrong.
With AI, we also need to design for what happens when everything goes right.
If an AI application takes off, the customer won’t just need more compute. They’ll need somewhere for all that success to live.
