IBM i TRs Bring New Open Source Data Transfer Tools
February 22, 2017 Alex Woodie
IBM i shops that adopt the latest Technology Refreshes from IBM will find a trio of new open source tools for moving data and objects across various data transfer protocols used on the Internet. The addition of Wget, cURL, and rsync to the IBM i Open Source solutions package will give IBM i shops more choices for moving data.
As IBM added open source development tools like Python, PHP, and Node.js to the platform, it has triggered requests for additional tools that open source developers are familiar with, says Alison Butterill, the offering manager for IBM i in Rochester, Minnesota.
“These three tools Rsync, Wget, and cURL, are very popular with open source programmers,” Butterill told IT Jungle‘s Dan Burger last week. “Developers requested these be put into the system.”
So what does IBM i get with Wget? According to the GNU Project, Wget is a free utility for non-interactive download of files from the Web. A user can use Wget to set up a download from a remote system, and then walk away from the computer, knowing that Wget will handle the download by itself, and then terminate the session when it’s done.
Wget supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies, and was “designed for robustness over slow or unstable network connections,” GNU says. “If a download fails due to a network problem, it will keep retrying until the whole file has been retrieved.” And if the remote server supports “regetting,” Wget will instruct it to continue the download from where it left off.
The developers of Wget, Giuseppe Scrivano and Hrvoje Nikšić, originally designed the software the 1990s to fill a gap in Web downloading space, since no single program could work over HTTP and FTP. It became immediately popular among Unix users who were struggling with slow dial-up lines. Since then, it has been ported to run on Linux, Windows, Mac OS, OpenVMS, AmigaOS, MorphOS, HP-UX, and, now the IBM i OS.
Here’s a fun fact about Wget, per the tool’s Wikipedia entry: “In 2010, US Army intelligence analyst PFC Chelsea Manning used Wget to download the 250,000 US diplomatic cables and 500,000 Army reports that came to be known as the Iraq War logs and Afghan War logs sent to WikiLeaks.”
Meanwhile, cURL (not to be confused with the scripting language Curl) is another tool for moving data and objects over the Internet. The software, which is a GitHub project, provides a command line and a library for transferring data with URL syntax over a wide variety of protocols.
The cURL (which means “see URL”) supports the libcurl library, which allows it to support HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, TELNET, DICT, LDAP, LDAPS, FILE, IMAP, SMTP, POP3, RTSP and RTMP, according to its GitHub listing.
According to IBM, one of the big benefits of cURL is that it supports advanced features, like proxies, HTTP POST and GET requests, FTP upload, and user authentication, the company says in its IBM i 7.3 TR2 announcement letter. (All three of the open source solutions discussed here are available with IBM i 7.2 TR6 as well.)
IBM i developers adopting open source programming languages drove the addition of cURL, says Jesse Gorzinski, business architect for IBM i open source. “Some of the Python libraries that do cURL-like things rely on cURL to get it done,” he told Burger. “One of the things we have done is enable the Curl-like things in Node.js and Python that rely on the underlying Curl library.
IBM has also modified its open source Git offering to support cURL. Git, of course, is the distributed version control system created by Linus Torvalds a dozen years ago to manage the development of Linux, and which IBM brought to IBM i last year with the launch of IBM i 7.3. Adding cURL support to Git (found in Option 6 of 5733-OPS) allows for the interaction of remote repositories with HTTP addresses, IBM says.
Last but not least is the addition of rsync to both IBM i 7.2 and IBM i 7.3 (no more soup for you, IBM i 7.1!). Another Unix-era utility, rsync is a utility for efficiently transferring and synchronizing files or entire directories from one computer to another.
According to its Wikipedia entry, the utility uses a delta encoding algorithm to minimize network usage. If the receiving computer has a file that’s similar to, but not identical, to the file being sent, it will employ an algorithmic method to determine what to send. It uses a combination of an MD5 hash and a rolling checksum hash to determine exact differences between the two files.
Originally released in the late 1990s by its original developers, Andrew Tridgell and Paul Mackerras, rsync today can be found in all Linux distributions, and has since been ported to Windows, FreeBSD, NetBSD, OpenBSD, MacOS, and, now, the IBM i OS. It’s distributed under a GNU General Public License (GPL). The software, which is renowned for its speed, flexibility, and script-ability, has also found its way into a number of commercial products.
All of these products were added to Option 7 IBM i Open Source solutions (5733-OPS).
RELATED STORIES
IBM i Open Source Business Architect Lays Out A Plan
7 Must-Have Open Source Products for IBM i
MariaDB Dropping In On IBM i To Replace MySQL
