Eradani Lays Waste to API Payload Restrictions
November 15, 2023 Alex Woodie
Eradani Connect customers who need to pass large amounts of data to IBM i via external API calls have often been stymied by a built-in restriction in IBM i that limits their payload to a mere 16MB. Those customers will be thrilled that Eradani has launched a new big data connector that smashes those payload restrictions to smithereens.
Eradani Connect is a packaged application designed to simplify the creation and execution of API calls on IBM i. The software functions as a middleware layer that makes it easy for .NET, JavaScript, and Python developers to get access to the IBM i via API calls.
Flexibility is a key trait with Connect. It supports REST- and SOAP-based protocols. XML, JSON, and CSV are all supported data types. And of course it supports multiple data transportation methods too.
Customers looking to tap into IBM i applications and data have a choice of using protocols like ODBC, FTP/SFTP, pub/sub brokers (i.e. Kafka), iDB, and XMLService, along with a bulk data connector. Many customers choose XMLService since it’s easy to use and flexible, Eradani says.
“If you know how to use it, XMLService can be quickly configured and called from virtually any programming language, which makes it a great choice for experienced users who need to quickly set up integrations on the IBM i and whose programs have relatively small parameter signatures,” Eradani CEO Dan Magid tells IT Jungle.
However, there’s a downside to using XMLService: it’s limited to passing 16MB at a time. That 16MB limit is due to the fact that XMLService stores data in a fixed-character field on IBM i, and a fixed-character variable in RPG maxes out at 16MB.
Now, if customers could get the full 16MB payload out of XMLService, it might not be such a dealbreaker. But the reality is that the API payload when using XMLService often ends up being much smaller than 16MB due to all of the other stuff that needs to be sent, including all the configuration options for each parameter the customer is passing.
“I’ve seen calls that couldn’t even pass in 1 MB of data to their core programs before hitting the data limit in XMLService because of all the XML data overhead,” Magid says.
XMLService works well for programs that have just a few parameters. But Eradani customers often have programs with tens of thousands or even millions of total inputs. One of Eradani’s customers was trying to pass 5 million total inputs via XMLService, which took 13 minutes to allocate all of the data values for that program call, Magid says.
Eradani Connect supports an array of API protocols, data types, and IBM i connector options (Image source: Eradani)
Clearly, waiting 13 minutes for an API call to process is not ideal, so Eradani went back to the drawing board. What it came up with was a new big data connector, which it named, appropriately enough, the Big Data Connector.
Magid explains how it works:
“Eradani’s Big Data Connector works by generating specialized code to handle the parameter translation between languages rather than going through a generic connector like XMLService. As a result, it is able to optimize the translation for each specific use case and take advantage of additional features such as data compression and encryption.”
When Eradani applied the Big Data Connector to the API calls of the customer with 5 million total inputs, the processing time from start to finish dropped to quarter of a second. “And that’s including the network latency,” Magid adds.
Eradani has also added a new EDI connection to Connect, giving customers another option for data transport. EDI, in its various protocols, continues to be widely used in the consumer goods and automotive supply chains.
Giving customers choice and flexibility is part of the Eradani equation, Magid says.
“At Eradani, we’re big proponents of using the right tool for the job. XMLService has some great strengths which is why we still offer connectors in our suite that use XMLService — it’s a great tool for its intended use cases,” he says. “As with all technologies, however, it has its drawbacks and its specialized use cases. Larger amounts of data are not a good case for XMLService.”
Just as RPG is great at some things and not-so-great at others, XMLService has its strengths and weaknesses.
“There are some use cases RPG was designed for and some it wasn’t designed for,” Magid continues. “Parsing large amounts of XML data, acting on dynamic instructions, and using dynamically allocated memory are all things XMLService has to do, but which RPG isn’t suited for. Using RPG for tasks it wasn’t designed for leads to limitations like this.”
For more information on Eradani Connect’s new Big Data Connector, check out its website at www.eradani.com.
RELATED STORIES
Eradani Debuts DevOps Suite for IBM i
