Using FTP to Poll for File

April 6, 2005 Hey, David

I am moving data between an iSeries system and several Unix systems. I pull the files over manually using FTP and then use the Copy from Import File (CPYFRMIMPF) to copy the records to a database file. After processing is done, I send the files back to the Unix system.

The biggest “hole” in the process is that there is no automated way to bring over the files from the FTP server onto the iSeries. I would like to have a polling process that would go through a list, check for new files to process, automatically run the FTP GET, and kick off a batch job to process the file. Do you have anything like that?

–Mark

After receiving Mark’s initial email, I found out some more details that helped to shape the solution described below. Those details helped me understand why a commercial solution wasn’t an option and that a Java-based solution might have advantages in Mark’s mixed system environment.

What I sent Mark was a program that checks for incoming files residing on other systems and pulls those files down to the local system. I named that program FtpPoll and if you are interested, you can download the source and follow along while I describe how this program works.

The purpose of the FtpPoll program is to poll any number of servers checking for files to retrieve. When a file is found, a thread is started that retrieves the file to the local server via FTP. After downloading the file to the local server, the FtpPoll program deletes the file from the remote server. This process repeats at regular intervals until a shutdown request arrives in the control data queue.

The main processing occurs in the pollFile method. That method checks for files to download and then waits on a data queue for the number of seconds specified in the POLL_INTERVAL_SECONDS variable before checking again. If an entry arrives on the data queue, the polling process ends.

The addFtpFile method adds remote files to the list of files that the program polls. An inner class of FtpPoll named FtpFile defines file attributes like the FTP URL, user, password, remote file name, and local file name and is passed to addFtpFile. For testing purposes, the main method creates an FtpFile if five parameters representing FTP URL, User, Password, file, and the local file name to download to and adds it to the ftpFiles set. Of course, in most cases, you would load the ftpFiles array from a database or hard-code the files in the main method. Since the ftpFiles variable is a java.util.Set, duplicates are not allowed based on the compareTo method in FtpFile.

When a file is located on a remote server, the pollFile method calls the retrieveFile method. The first thing the retrieveFile method does is start a new thread to perform the actual download. This allows control to return to the pollFile method immediately. When the thread starts, a sending flag is set to “true” to prevent simultaneous retrieval requests for the same file. The file is then retrieved to the local server and then the remote file is deleted.

You will need to do some setup before you can run this program on your iSeries system. First, if you are not currently running production Java programs on your system, I recommend that you get your system up to date on PTFs. That means you should load the latest cumulative and group PTF packages. If possible, you should also make sure you are using version 1.4 of the Java Development Kit on your system. You can verify typing in GO LICPGM from an OS/400 command line. Select option 10 to see if licensed program 5722JV1 option 6 (the JDK 1.4) is installed. If it is not installed, do so by selecting option 11 and placing a 1 next to 5722JV1 option 6.

After verifying that your JDK and PTFs are up to date, save FtpPoll.java in an IFS directory and start Qshell using the QSH command. The comments at the top of the FtpPoll.java source describe the commands you need to use to set your class path, compile, and run FtpPoll. Before testing on live data, remember that the remote file is deleted after it is downloaded.

I hope FtpPoll gets you off to a good start. In its current state, it is pretty basic. But I am here to help if you run into trouble adapting it to your environment or need to add additional features.

–David

Click here to contact David Morris by e-mail.