Before 2023, cloud infrastructure was mostly a topic for technical teams in many companies. In the framework described by Forrester, the cloud was usually discussed around cost and modernization topics. With ChatGPT, this picture has changed. The cloud is no longer only a technical choice. It has become one of the main topics in business strategy. The billion-dollar initiatives of giants like NVIDIA, Microsoft, and Oracle clearly show this. However, this growth has also made us face a different reality. Generative AI is shaking the cloud’s promise of being unlimited, cheap, and pay-as-you-go.
When I look at this issue from the perspective of a project manager, I think the problem is bigger than a technical choice. Because whether you place artificial intelligence in the cloud, on-premise, or in a hybrid structure directly affects the project’s cost structure, risk profile, maintenance load, and long-term sustainability.
Why is the cloud still a strong option?
There are very clear reasons why the cloud looks strong for generative AI. Especially in the first trial stage, starting quickly, avoiding new hardware investment, and moving forward through ready-made services offer important advantages. Data from WebbyLab and SoftwareSeni shows that a cloud-based AI project can be launched at the MVP stage with a starting cost that is 5 to 10 times lower than a local setup. For this reason, many companies take the first step in the cloud.
In addition, the cloud provides a genuinely powerful model for workloads that fluctuate. When the usage level of a product is still uncertain, when the team is trying out different scenarios, or when rapid prototyping is the aim, the cloud’s flexibility delivers real value.
When does the problem begin?
The real breaking point happens when the system moves from being a temporary trial to a continuously running structure. According to SoftwareSeni, AI costs can increase by 5 to 10 times only a few months after the system goes live. A structure that looks economical at the beginning can turn into serious budget pressure in continuous use.
This is no longer an exceptional example. According to Barclays data, 83 percent of businesses are planning to move workloads back from the public cloud to private infrastructure. According to Cloudian’s 2026 report, 93 percent of companies have either already moved their AI workloads to local infrastructure or are actively considering it. These data points show that the cloud repatriation approach has become a financial and operational decision.
The 37signals example also makes this change clear. The company says that by leaving the cloud and moving to its own hardware, it saved 7 million dollars in 5 years. According to an Andreessen Horowitz analysis, high cloud spending can reduce the gross margins of a software company by 50 percent. Moving some workloads to on-prem infrastructure could theoretically be effective enough to double the company’s value.
The hidden costs of artificial intelligence
In AI projects, people often talk only about the license, API, or cloud bill. But the real cost items are often hidden inside the architecture. According to NovoServe data, data egress fees can go above 10,000 dollars per petabyte. So even if moving data to the cloud is relatively easy, taking it back out can become a serious financial barrier.
GPU costs are reshaping the picture in a similar way. Running 64 NVIDIA H100 GPUs in the cloud comes to roughly 800,000 dollars per year, while on a local server that figure can drop to about 400,000 dollars, depreciation included. This gap explains why local infrastructure is back on the agenda, particularly for AI workloads that run continuously.
Data sovereignty is another major topic that is growing this discussion. Auvik data shows that 20 percent of enterprise workloads are moving to local data centers because of legal and sovereignty concerns. This shows that where data is physically stored is no longer only a technical issue, but also a managerial and strategic one.
When does the cloud become more expensive than on-prem?
For most companies, there is still no final choice like full cloud or full on-prem. The 60 to 70 percent continuous usage threshold presented by SoftwareSeni offers an important reference here. According to this approach, if your system is active regularly and for a long time during the day, after a certain point the cloud becomes more expensive than on-prem. On the other hand, for sudden demand increases, model training, or temporary heavy usage, the cloud still offers an important advantage.
For this reason, the hybrid model is becoming more and more logical. Keeping the base workload fixed on local servers and moving sudden capacity needs to the cloud creates a more balanced model in terms of both cost and flexibility. Tools such as Azure Arc, AWS Outposts, and Rafay also increase the manageability of this hybrid structure.
The questions to ask before deciding
Today, I do not believe the first question companies should raise when deciding is, “Is the cloud more modern, or is on-prem more secure?” Instead, a decision can be reached by first asking questions like: How continuously will the workload operate? What type of data will it handle? Which regulations will apply to it? And what will the total cost be over the long term?
Data from different sources such as Barclays, Cloudian, 37signals, Andreessen Horowitz, NovoServe, Auvik, SoftwareSeni, and Alithya shows that there is no single correct answer in AI infrastructure. The cloud is still a strong option for speed, experimentation, and innovation. Local infrastructure and hybrid models, on the other hand, are again being evaluated more strongly in terms of cost control, data sovereignty, and continuously running workloads. For this reason, it seems more meaningful to look at this issue not only as a technical preference, but as a decision area that should be handled according to the structure of the workload, the quality of the data, and long-term sustainability.