Getting A Handle On What GenAI Might Cost
August 21, 2024 Timothy Prickett Morgan
Because of my other day job at The Next Platform, I can give you a pretty good idea about what it costs to train an AI model or run inference against it once it is trained. But I have seen very little data that tries to give any of a sense of what it costs to add generative AI functionality to applications.
But interestingly, Gartner analysts embedded some such data in a report about how it expected many GenAI projects to be abandoned by the end of 2025 after their proofs of concept fail. That so many PoCs will fail – the estimate was for 30 percent of projects to be shut down without success – is not a surprise to us. This is often the case with innovative IT projects. In fact, we are surprised that it will not be 50 percent – or higher, given the extraordinary hype cycle we are all experiencing right now.
These GenAI projects are going to fail for a lot of reasons, according to Gartner, but poor data quality, inadequate risk controls, escalating costs, and unclear business value are the big reasons.
“After last year’s hype, executives are impatient to see returns on GenAI investments, yet organizations are struggling to prove and realize value,” explains Rita Sallam, distinguished vice president analyst at Gartner, who put together the report. “As the scope of initiatives widen, the financial burden of developing and deploying GenAI models is increasingly felt.”
Coming up with the money to do these GenAI projects is a major barrier, which is why Gartner came up with some figures for GenAI projects in the first place. Take a look:
As you can see, code assistants are relatively cheap, and this is where we expect for many organizations to take their first steps in GenAI for this reason. IBM is working on an RPG coding assistant right now and we suspect that others in the application modernization field will try to take a stab at it, too. Getting RPG code to train a model that can generate reasonably good RPG code could be problematic. It will take a lot of code to do so. But IBM i shops might end up using a code assistant for Java to start, should one be created. IBM is also working on a code assistant one to convert mainframe COBOL to Java, but not one to generate pure Java from scratch.
For typical customers, this could cost a couple hundred thousand dollars to add to the programming toolchain. This may sound like a lot, but other GenAI projects, as the table above shows, will cost millions of dollars and thousands to tens of thousands of dollars per year per user to maintain. Obviously, the benefit of the above project has to be greater than the cost of the GenAI implementations and add-ons to the applications.
If it is tricky to figure out what GenAI functionality costs, we think it is even harder to figure out what effects GenAI projects will have on cutting costs and boosting revenues, or just making workflows better or easier. Every company is going to have to try to figure that out ahead of time, even before they do the GenAI PoC, so they have a target to aim at and to benchmark reality against. Even if 70 percent of GenAI projects “succeed” – meaning they do not literally fail on a technical basis – that does not mean that they will be economically beneficial to the companies that do them.
This is not as obvious as putting a storefront and a web page on the Internet. But then again, maybe back in the late 1990s the Internet was not as obvious as it now feels, either.
RELATED STORIES
Some Thoughts On Big Blue’s GenAI Strategy For IBM i
How To Contribute To IBM’s GenAI Code Assistant For RPG
IBM Developing AI Coding Assistant for IBM i
The Time Is Now To Get A GenAI Strategy
Top Priorities in 2024: Security and AI
IBM i Shops Are Still Getting Their Generative AI Acts Together
Generative AI Is Part Of Application Modernization Now
Sticking To The Backroads On This Journey
With Fresche’s New CEO, There Are No Problems, Just Solutions
IBM Introduces watsonx For Governed Analytics, AI
Technology Always Replaces People While Augmenting Others
GenAI is still in a super hype phase due to investment and promises, it needs to be understood in its realistic limits and deflate a little and prove in industrial fields, and value / $.
IMHO in software a see a space in “error checking”, “best practice”, “suggestions” integrated in IDE, besides pure generation and templating.
Still, the energy costs of these genai clusters are incredible and massive (ain’t this a “green economy”?), carbon foot print is incredible (no, buying CO2 paper offset titles doesn’t count to feel better) and pales in comparison to the human brain in term of efficiency and results, that does task using tens of Watts.
The risk of AI feeding itself is there when more and more content outside is generated by AI, and after a few iterations will produce nonsense, I don’t even imagine the consequences, yet to be explored….