Volume 7, Number 13 -- April 3, 2007

Thoughts on the Coexistence of Full Test Automation and Manual Testing

Published: April 3, 2007

by Colin Armitage

Automated software testing solutions have been around for more than 20 years but successful implementations are few and far between. Most organizations continue to execute the majority of their testing manually. Industry analysts peg market proliferation at just five percent, with 30 percent being the most extreme estimate. One reason for this damning failure is that users were promised that test automation could address 100 percent of their quality assurance requirements, but bitter experience has shown that total success was far more elusive.

For these reasons, it can be said that anyone who tells you that software testing can be totally automated is lying.

Expectations are a big part of the problem. The reality is that full automation is not always appropriate and the amount of effort necessary to achieve full automation should never be underestimated.

There will always be a trade off between developing a fully automated process and just getting by with the manual testing. The key metric in the discussion of whether it is wiser to automate or not automate is re-use. It will always take longer to develop an automated test bench but there can be overwhelming payback if tests can be executed multiple times. If this "economy of scale" business case cannot be made, then organizations will always logically choose to execute their testing manually.

In real life, the situation is very complicated since, for example, any third party application that has undergone extensive modification could be a moving target for testers. With new releases and bug fixes always in the queue, it's nearly impossible to nail down a stable version to test. If an automated solution cannot adapt to a forever-changing application, then the potential for re-use simply disappears, along with the justification and user enthusiasm for automation.

On the other hand, the great thing about automation is that it enables types of testing that are practically impossible to approach manually. Simulating extreme workloads and logging a program's response can help you avoid large-scale problems once your application goes live. To attempt to test the breaking point of an application manually, dozens of developers or users would be needed for weeks or months and this is simply not practical.

Regression testing is the process of comparing the results of initial baseline tests with those of current tests. The object of regression testing is to ensure that areas of an application are unchanged by a new release and continue to work as intended. Full regression testing can provide vital business stability throughout application changes but is incredibly difficult to perform manually due to the time, resource, and knowledge constraints.

Life would be so much easier if the barriers to develop automated process could be lowered. All the traditional solutions require in depth programming skills that are rarely found among the quality assurance (QA) fraternity, whose strongest members come from the business side and are subject matter experts, rather than technical gurus.

Even if testers can be persuaded to embrace such skills, it can be difficult to find time for the development of test scripts, test the scripts themselves, and train legions of testers in their use, while continuing to make progress with on-going test projects.

With seemingly no clear or simple way to transition from a manual testing protocol to one that avails itself to the benefits of automation, people grow confused, lower their heads, and continue with the old way of doing things. In doing so, they lose out on automation's important benefits: Principally the ability to address one of modern business' key challenges, which is how do I reduce application time to market and thus deliver real world competitive advantage?

You can solve most problems by first adopting a good strategy.

To gain the commitment from the business and test teams to improve QA it is essential that the potential benefits from automation are aligned to effort and time expenditure, and can be incrementally delivered when the business requires. I call this approach Crawl, Walk Run.


We all know the importance of an effective plan that enables you to take control and manage the whole testing process. The ability to manage and report progress against that plan is becoming a business imperative due to the ever increasing compliance and audit pressures modern businesses are facing today.

As already discussed, a significant portion of testing is and always will be manual, so let's start there. Hopefully you are already recording your manual testing along with any defects that are discovered. The challenges you face include reproducing defects, constructing test audit trails and verifying the results at visual and database levels.

The last six months have seen some exciting developments in this area, not least the announcement of assisted manual testing solutions that can automatically document and track your steps while testing in a non-intrusive fashion, allowing you to correct defects much quicker, by re-creating the problem at a touch of a button.

With the initial successes and quick ROI enjoyed in this Crawl phase, you will have more time to continue process improvements which will in turn allow you to benefit even more.


Once the testing fundamentals of planning and manual testing have been addressed you should be in a position to move to the Walk phase, which for many companies revolves around the subject of managing test data. What does managing test data involve? In a nutshell it is the creation, management, and re-use of data in a secure and protected environment. Test data management is a cornerstone to successful testing and is an area where technology can often delivery a quick payback.

Effective test data creation provides more practical test verification and it also addresses issues relating to disk space, data verification, data confidentiality and protracted test cycles. Companies that do not manage their test data properly will always find any testing automation very painful, and verification almost impossible.


With the fundamentals in place and a strong hold on your test data, you are in a powerful position to implement full test automation in every area it can be justified. We have already seen that effectively deployed software automation solutions can have a significantly positive effect on functional and regression testing timescales. Implemented correctly, automation can offer great benefits in areas such as time to market, risk reduction, cost savings, and resource efficiencies.

Colin Armitage is CEO at Original Software, a company that specializes in the development of automated software testing solutions. Armitage is a renowned expert on software quality control.

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot

Sponsored By

Customize your System i education in 2007 with
COMMON's 2007 Annual Conference & Expo,
April 29 - May 3, 2007 in Anaheim, California.

This premier System i education and networking event promises to be larger than our recent conferences, providing invaluable learning and networking opportunities for you and your employees.

The Annual Conference & Expo will be the flagship event of the new "COMMON…Customized" education model, offering over 500 educational sessions, hands-on labs, and all-day workshops. We've added new tracks on hot topics like PHP, open source, and IP telephony - all delivered by the most respected experts in the IT industry. In addition, the new COMMON Annual Conference & Expo will offer non-technical, professional development sessions to help IT managers resolve daily business issues.

COMMON's 2007 Annual Conference & Expo will offer:
· More session hours, including over 500 sessions and hands-on labs
   in a range of choices every hour.
· More user sessions that are driven by users, including customer experience sessions.
· More in-depth education that includes all-day pre-conference workshops,
   all-day Integrated Seminars, open labs and a wide variety of regular-length sessions.
· Emphasis on networking that provides great opportunities to network with your peers,
   IBM developers and executives, and industry experts.
· An extensive Expo of new companies showcasing the latest System i-related
   industry solutions.

The COMMON Annual Conference & Expo is one of the most cost-effective ways to gain knowledge and tools needed to meet the changing demand of IT. You'll pay a reasonable amount for intensive education, and realize a tangible, immediate return on your investment. In addition to the direct cost savings, the networking opportunities and professional contacts are immeasurable.

The COMMON 2007 Annual Conference & Expo is a System i educational and networking event that you and/or your team won't want to miss. To learn more about the conference and to register online, visit www.common.org/conference.html.

The Annual Conference & Expo is just one of the many benefits to COMMON membership, so don't miss out on your opportunity to attend this premier event.


Editor: Alex Woodie
Contributing Editors: Dan Burger, Joe Hertvik,
Shannon O'Donnell, Timothy Prickett Morgan
Publisher and Advertising Director: Jenny Thomas
Advertising Sales Representative: Kim Reed
Contact the Editors: To contact anyone on the IT Jungle Team
Go to our contacts page and send us a message.

Sponsored Links

VAULT400:  Securely archive data with Instant Back-Up & 24x7 Recovery
Bsafe:  Enterprise Security for System i and Linux Systems
COMMON:  Join us at the 2007 conference, April 29 – May 3, in Anaheim, California

IT Jungle Store Top Book Picks

The System i Pocket RPG & RPG IV Guide: List Price, $69.95
The iSeries Pocket Database Guide: List Price, $59.00
The iSeries Pocket Developers' Guide: List Price, $59.00
The iSeries Pocket SQL Guide: List Price, $59.00
The iSeries Pocket Query Guide: List Price, $49.00
The iSeries Pocket WebFacing Primer: List Price, $39.00
Migrating to WebSphere Express for iSeries: List Price, $49.00
iSeries Express Web Implementer's Guide: List Price, $59.00
Getting Started with WebSphere Development Studio for iSeries: List Price, $79.95
Getting Started With WebSphere Development Studio Client for iSeries: List Price, $89.00
Getting Started with WebSphere Express for iSeries: List Price, $49.00
WebFacing Application Design and Development Guide: List Price, $55.00
Can the AS/400 Survive IBM?: List Price, $49.00
The All-Everything Machine: List Price, $29.95
Chip Wars: List Price, $29.95


The Four Hundred
Next Up on the System i5: Native GNU g++ and IBM XL C/C++

WDSc V7.0: Componentization of Advanced Edition Is Not Enough

Gartner Says It Was "All Over" the Virtualization Effect

Kronos To Be Taken Private Through a $1.8 Billion Buyout

The Linux Beacon
Novell Previews SLES 10 Service Pack 1, OES 2 Extensions

IDC Chops Server Forecasts Thanks to Virtualization, Multicore Chips

Supermicro Shows Off Blades, Posts Financials Ahead of IPO

NEC, Stratus Preview Fault Tolerant Server with Quad Cores

Big Iron
IDC Chops Server Forecasts Thanks to Virtualization, Multicore Chips

Top Mainframe Stories From Around the Web

Chats, Webinars, Seminars, Shows, and Other Happenings

Four Hundred Guru
Two Views on the WDSc Snippets View, Part 1

The Case of the Missing Outline (View)

Admin Alert: Five Things that Kill Backups (and What to Do About Them)

System i PTF Guide
March 24, 2007: Volume 9, Number 12

March 17, 2007: Volume 9, Number 11

March 10, 2007: Volume 9, Number 10

March 3, 2007: Volume 9, Number 9

February 24, 2007: Volume 9, Number 8

February 17, 2007: Volume 9, Number 7

The Windows Observer
Microsoft Ships Operations Manager '07, Taps EMC for Network Monitoring

Windows Vista Sales Are Hot, Hot, Hot! Microsoft Says

Oracle Sues SAP Over 'Corporate Theft on a Grand Scale'

NEC, Stratus Preview Fault Tolerant Server with Quad Cores

The Unix Guardian
Sun Breaks Sparc Unit Free Again

Gartner Says It Was "All Over" the Virtualization Effect

Oracle Sues SAP Over 'Corporate Theft on a Grand Scale'

As I See It: Workplace Heaven

Four Hundred Monitor
Four Hundred Monitor's
Full iSeries Events Calendar


Vision Solutions
Patrick Townsend & Associates
RJS Software Systems

CYBRA Finds the 'Edge' for Native i5/OS RFID Software

Lakeview Adds More Autonomics to MIMIX

Thoughts on the Coexistence of Full Test Automation and Manual Testing

Help/Systems Boosts Graphics with Robot/NETWORK V10

News Briefs and Product Shorts:

Infor Buys Workbrain for About $200 Million . . . Quadrant Addresses Canadian Check Requirements . . . IBM Delivers New Release of RFID Software . . . Esker Launches Fax Services for SAP . . . Make That Three 5250 Emulators Supporting Vista . . . ProData's Hartley a Split Second from Milestone Victory . . .

Four Hundred Stuff


Subscription Information:
You can unsubscribe, change your email address, or sign up for any of IT Jungle's free e-newsletters through our Web site at http://www.itjungle.com/sub/subscribe.html.

Copyright © 1996-2008 Guild Companies, Inc. All Rights Reserved.
Guild Companies, Inc., 50 Park Terrace East, Suite 8F, New York, NY 10034

Privacy Statement