Development Horror Stories Surface as Aldon Unveils Turkey Award Winners
December 16, 2008 Alex Woodie
Some people watch car racing strictly for the high-speed accidents. (We know you’re out there.) Others take particular glee in seeing live performers melt down on stage. (You may be one.) When it comes to IT, there are equivalent debacles, but they’re normally shared in whispered tones after work at the local pub. Now, thanks to Aldon, which teased the worst sort of tales out of AS/400 developers in return for the chance to win prizes, IT Jungle readers get to hear some of them. And frankly, the stories do deliver. (Not that you’re into that kind of thing.)
Before the Thanksgiving holiday, the good folks at Aldon Software announced a competition like none other: the Developer Turkey Awards, where you recount how a programming mistake that you made (or a “friend” made) nearly brought your company (or your “friend’s” company) to the brink of ruin.
Two categories were presented: the Most Frightening Development Nightmare, which is sort of the “unlimited” class of top-fuel IT catastrophes, and the Most Remarkable Project Turn-Around, which is where source code disasters are narrowly avoided but for the quick application of Aldon software configuration management (SCM) solutions.
For obvious reasons, the Most Frightening Development Nightmare is where we find the most titillating, R-rated AS/400 project meltdowns. There is also good stuff in the Most Remarkable Project Turn-Around category, but they are PG-rated stories with happy endings by comparison. To hear how that programming mistake went from “Phew, close call” to “Run! It’s gonna blow!” you gotta go with the Nightmare division.
And the winner of the big prize for Most Frightening Development Nightmare goes to Brad Abernathy, a 20-year veteran in development. Here’s how Aldon describes Abernathy’s experience:
“His story details how one accidental coding override led to total chaos, and almost a million dollar mistake! Back when Abernathy was a developer for a major manufacturer of high-end bedding and towels, his colleague was working on a project involving the shipping/re-order application. That developer created a logical file but did so right on top of the production physical. This created a bug that turned product order quantities from just two or three towel bundles into two or three million! Because there was no quality check, the systems nearly blew up trying to calculate and distribute orders in such massive quantities. It took several days and lots of overtime to get back to ‘business as usual.'”
“This was hands down the worst incident I have ever seen as a result of no source or object management,” Abernathy told Aldon. “It was definitely a turkey disaster that could have been prevented if we had a change management solution like Aldon.” Not surprisingly, Abernathy uses Aldon at his current job, Sunbelt Rentals.
The runner-up winner in the Nightmare division is Nathan Hoy, who previously worked in the IT department of a major retailer (who will remain nameless). Here are the gory details of Hoy’s nightmare, straight from Aldon:
“Hoy was working with a newly hired developer who wrote a program to purge a very large file–one with more than 57 million records. When that developer accidentally named the purge program the same name as a very active purchase application, chaos rippled through the retail sales channel, as point-of-sale machines froze up and customer tempers flared.”
Today, Hoy is a senior program analyst with L.L. Bean, a long-time AS/400 shop that was ranked the number one online clothing store by Consumer Reports magazine this year. Obviously, that company has its act together (in the IT department and elsewhere), and its use of Aldon’s SCM software helps to prevent that sort of mistake from occurring. Hoy also installed Aldon at his previous employer following the screw-up.
The winner in the Most Remarkable Project Turn-Around category is Manoj Dhamu, a senior programming analyst with DST Health Solutions, developer of OS/400-based health plan administration and claims processing software, and an Aldon customer. We’ll let Aldon tell the rest:
“Dhamu and his team were using Aldon to manage its development–from build to deployment–but ran into a major glitch when they realized their product plays host to numerous custom programs with source code outside the area where Aldon was installed. With a looming deadline and the realization they would have to manually run those custom programs every time they rebuilt a file, the pressure was on. Fortunately, Aldon was able to re-tool the solution and incorporate all of this important external code into the development cycle. What could have taken months only took days–which meant tremendous cost savings for DST Health Solutions.”
The runner-up winner in the Turn-Around category is Norman Martens, a development architect for a major health care services provider. “He credits Aldon for helping ensure the company’s primary warehouse application–which is considered one of the organization’s major lifelines–never went down during a year-long application update with more code moving around than you could shake a stick at,” Aldon tells us.
Martens adds: “Thanks to Aldon, when the auditors come, all we need to do is point and click to a specific report to fulfill compliance requirements. Now that is what I call avoiding disaster.”
The two winners get a Nintendo Wii video game system, while the runners-up get an Apple iPod Touch. All four winners received a juicy holiday turkey shipped right to their homes, as well as a charity donation going out in their names. And the rest of us get the chance to hear their horror stories–and be thankful that it wasn’t us!