DataMystic Provides iSeries Data Transformation Services

DataMystic has a new release of TextPipe, a Windows utility that lets users do all sorts of things to text files, including cleaning data in preparation for a data warehousing project and translating between ASCI and the EBCDIC formats. Version 7.5 bolsters the product’s OS/400 and mainframe story by bringing new report conversion capabilities through its “mainframe copybook wizard,” and expands its foreign language translation support.

DataMystic is a software vendor based in Melbourne, Australia, that has been developing the TextPipe product since the company was founded 15 years ago. The company, which changed its name from Crystal Software earlier this year (there are a lot of different Crystal Softwares out there, and the company wanted to differentiate itself), has more than 2,000 customers across the world, including companies in the banking, telecommunications, healthcare, manufacturing, legal, and publishing industries.

DataMystic’s TextPipes product is based on the “pipes” concept used in Unix and DOS operating systems. Basically, pipes enables a program to run, and for its output to be fed into the next program as its input. TextPipes expands on this concept by breaking each file up into small pieces, which are progressively fed through a series of pre-defined or custom-written filters. Each filter performs a simple piece of text manipulation, like a search and replace, converting to uppercase, or translating between languages or encodings.

When the first filter is finished with its piece, the result is fed into the next filter, and finally the output file. Alternatively, users can set up the product’s sub-filtering, or two-dimensional text processing, capability to enable allows sub-sections of the text to be operated on independently of the main text. This architecture enables TextPipes transformations to run quite quickly, with no limitation on file sizes or the number of operations performed on each file, and minimal impact on memory and disk, according to the company. In one case, TextPipe was able to make 17 million replacements in a 250,000-record, 75MB file in under two minutes.

TextPipe ships with more than 100 pre-defined filters, for items such as extracting the first name of a customer, fixing surname capitalization, or setting values to null, or users can build their own in either JavaScript or VBScript. The product enables users to implement changes with a fine degree of control, including making inserts according to a range of lines or columns, in a specific tab or comma separate value (CSV) fields, between HTML or XML tags, or even inside of custom tags. The product also includes something DataMystic calls its “EasyPattern” pattern matching language, and its associated EasyPattern wizard, to make it easier to make global changes, particularly when Perl is used.

It’s no wonder DataMystic refers to TextPipe as the “Swiss Army knife” of data utilities, considering the range of data formats supported, including: HTML, XML, fixed length and delimited plain text files; Unix, mainframe, OS/400, and Windows end-of-line formats; ASCII, ANSI, Unicode, and EBCDIC files; EDIFACT, HL7, SWIFT and other structured formats; spooled print files; and structured and unstructured reports. Other common uses of TextPipes includes extracting e-mail addresses and eliminating the redundant ones, removing HTML to make a downloaded Web page more readable. Integration with Windows scheduling utilities can provide batch-like functionality.

With TextPipe version 7.5, DataMystic has enhanced the product’s mainframe copybook wizard to support the conversion of OS/400 and mainframe reports to comma-delimited formats. The mainframe copybook wizard now copes with OCCURS fields with fixed number of recurrences, according to the company. Other changes include the capability to add filters by dragging and dropping, or double clicking, in the All Filters view; new filters for converting HTML to text; new filters for Unicode compose and decompose actions; new code page conversions for Japanese Shift-JIS and Chinese BIG5 encodings; and new sub-filters.

TextPipe 7.5 is available now. The product is available in three editions, including Professional, Standard, and Carrier editions. TextPipe Pro, which supports the EBCDIC to ASCI conversions and the mainframe copy book functions, costs $299 per user. For more information visit www.datamystic.com.