By ^Author

Testing and QA tools and services are being modified to meet more generalized quality requirements, as well as to deliver much more potent and versatile results than testing alone. Automated testing is morphing into automated code inspection.

^Sidebars

Time is never an ally, but a ruthless adversary. No matter the people, the plan, or the project, it shows no mercy. Especially for application development projects, this is painfully true. There is constant pressure to meet delivery deadlines or end up dead in the water. In the application development life cycle, testing routines are in place to uncover and eliminate software defects, but they are frequently hasty. Testing often misses critical problems, an ever-present malady that the development community shares collectively. This is due to a combination of factors ranging from deficient testing experience and expertise, to insufficient testing facilities.

A hard lesson learned is that identifying errors prior to implementation essentially reduces cost in the long run. A harder reality is that the errors themselves may be costly, but so is testing. Still, time remains the ultimate obstacle.

Thanks in part to Y2K preparation, time had to stand still for a few moments, and organizations had to take the time to pause, reflect, repair, and sometimes retire their existing installations. These projects extended an ideal opportunity for IT divisions to invest in new equipment and essential "spring-cleaning." The chance to tidy up their technology gave IT shops ways to mitigate these quality questions and concerns.

For example, IT shops can leverage new and existing compliance technology, services, and tools. Besides taking advantage of supplementary budgets, IT shops have devoted themselves to an exhaustive process of planning, pruning, and testing code. This includes extensive reporting every step of the way.

With the rigors of the Y2K remediation effort, application quality assurance (QA) testing tools and services have found a new purpose and focus beyond 2000. The Standish Group has found that not only are testing and quality assurance tools and services being modified to meet more generalized application quality requirements, but they are also being driven in a new direction to deliver much more potent, complete, and versatile results than testing itself can provide. Automated testing is morphing into automated code inspection.

Inspection Tool Market
(In Millions)

Standish Group research indicates steady growth for the inspection tools market over the next few years.

Today, select organizations are beginning to take an active approach to integrating the inspection capabilities of current technology with the lessons learned from Y2K compliance testing methodologies. It proves to be an interesting marriage that The Standish Group believes will have value, usefulness, and stamina.

Anything that will help the defect dilemma is good news for IT organizations. The Standish Group estimates application bugs and defects are responsible for $85 billion per year of lost revenue due to system downtime.

Application Ailments

Applications can be contaminated for a number of reasons, but the biggest culprits are hasty development, hurried and/or insufficient testing, and legacy system revision. How to remedy application ailments introduced during development, while preserving the health of current installations, is a dilemma every IT executive faces.

In addition, as companies try to match speed with the blinding pace of business, they adapt their applications to satisfy the business needs. However, they frequently fail to take into consideration the consequences of making system modifications. Over time, these alterations can affect the framework of the entire system and leave many scratching their heads as to what they have created, or where an error may lay in wait.

When code is manipulated or additions are made without tracking or noting all changes, it makes it difficult to establish a common lexicon. Developer's that engage in poor coding practices increase the fragility of the software itself. Repairing code redundancies, removing dead-end paragraphs, and making date modifications can rival -- in complexity and commitment -- recreating the Sistine Chapel on a pinhead by hand.

Putting these factors into play prompts many questions:

• How can development shops improve the quality of critical applications and make them ready for primetime?
• Is there a way to prevent the introduction of defects before or during the production cycle?
• How can companies ensure the reliability of their existing applications or even understand their system's architecture?
• Will code changes actually introduce errors?
• If errors do exist, where do they reside?

It's enough to make the most stalwart IT executives run and hide in their Y2K bomb shelters.

But managing the quality of software change will continue to be a daunting task beyond 2000. For this reason, The Standish Group foresees a future for inspection gadgets. An inspection tool's range and depth of analysis is unparalleled.

Code Quality Inspection Provider Service Market
(In $Millions)

Demand for application quality verification services is growing. Standish predicts this market will near the $1 billion mark by 2003.

Their inclusion in the application development process helps streamline the way to success. They also serve a vital function in evaluating current installations.

The ultimate inspection tool ROI is saved software quality, but most importantly, saved time. Having access to inside information promotes application accuracy and helps mitigate risk. The Standish Group predicts that application inspection tools will gain favor and momentum as the technology matures. Despite market infancy, the tools have strong vital signs. The Standish Group sees value in inspection, from individual applications to entire systems. So does the market, where demand for application quality verification services is growing. This market is small, but it has strong growth potential, especially after the turn of the century. Standish forecasts that by the year 2003 the code quality inspection provider market will near the $1 billion mark.

Inspection Defined

Automated inspection is the process of performing in-depth, code-level analysis of an application by using automated tools and processes to review up to 100% of the source code. Code inspection can be instrumental in the application development life cycle because it can find defects that testing misses, without requiring the use of test cases. It can also play a pivotal role in reducing software development costs by identifying code abnormalities and aberrations prior to the testing phase.

Inspection is equally advantageous in maintaining legacy systems. Over time, undocumented changes and updates made to an existing application can be registered as defects. Inspection procedures can reveal where these changes exist, report the risk of defect, and locate redundant or "dead" code sequences. The depth of analysis that inspection can provide is powerful, effective, and profitable.

The Standish Group distinguishes "inspection" technology by the ability to perform extensive, multiple language code-level analysis. The grand design behind these tools and services is to enable a deeper understanding of an application's properties at the code level, and reveal the impact of any changes made to the code itself. The attempt is to conserve time and effort abating lengthy redevelopment projects. These tools are a hybrid of quality assurance, code remediation, and testing technology and methods. They share a common technology design in two primary areas:

Graphic depiction of the application environment. The tool can construct "blueprints" of an application's architecture. These "blueprints" diagram code characteristics and exhibit application contents. Some tools offer an expanded look at applications across an entire enterprise. The benefit of this approach lies in the ability to view code sequences and how they correlate, permitting the identification of code inaccuracies and redundancies. Users can examine the intricacies of related application elements and develop relationship models. These models profile elements in need of revision, uncovering unassigned variables and even infinite loops (and classic examples of "spaghetti" code).

These tools can also help to clarify business functions and rules embedded deep within the lines of code. Some tools supply a type of metric "glossary" for coding references. They can also generate comprehensive program error messages and the summary reports exposing code problem hotspots.

Impact and side-effect tracking. In examining application code structure, it is vital to understand the input/output logic flow. An inspection tool advantage is the ability to simulate the effect changes would have on code through impact analysis. Studying proposed changes may significantly reduce the chance of tangled code flow.

Monitoring and simulating program execution sequences can improve software reliability by exposing sections that need correcting, or which are corrupt. This type of inspection is especially important in a development environment, prior to implementation. Tracing the code paths exposes program anomalies, which can be dealt with before production. Impact analysis applies to identifying problems within legacy applications, most importantly. Recognizing and editing legacy code polluted by years of repairs and updates is key to streamlining business function. Inspection tools can track code-level edits graphically and display the outcome with relative ease and automation.

Code inspection is not a new concept. It has long been touted as a proficient way to identify faults and defects, thus ensuring and enhancing software quality. However, conventional code inspection methods entail manually sifting through thousands or even millions of lines of code to find defects. This can be a costly and laborious undertaking.

An inspection tool's range and depth of analysis is unparalleled.

Off-the-shelf tools provide benefits. Alone, however, they can be cumbersome. In existing systems, installations differ and a packaged tool may have to be significantly reconfigured to achieve maximum results. This can tie up in-house resources and personnel.

Inspection Evolution:
Power of Reasoning

Reasoning Inc., a software quality assurance service provider in Mountain View, Calif., is an example of an active participant in this evolution of inspection processes. The company originated as a supplier of QA technology and services. Its portfolio included testing tools and services that provided an analysis of application structure, verifying date compliance, identifying date-related elements, and change impact reporting (with emphasis placed on automated inspection of software for Y2K defects). Employing their existing software analysis experience and inspection savvy, Reasoning has developed an innovative approach to inspection. The company is helping to reinvent QA testing, planning, and accuracy by automating the process.

Reasoning's InstantQA service offers a new application inspection methodology. It is a comprehensive inspection alternative that is designed to automate the software inspection procedure. The goal is a timely and thorough software analysis that will identify critical software defects often missed by standard testing. Standard testing centers on application functionality and tends to overlook the importance of application structure. That is a counterproductive mistake. The Standish Group has found that examining application structure is imperative to maintaining software quality. Unlike other testing methodologies, the Reasoning approach focuses on an application's structure to achieve an understanding of the application's quality. This inspection practice, a variation of "white box" testing, helps identify defects that could limit application reliability.

Yearly Inspection Contracts Gain Favor

The Standish Group surveyed IT executives from Fortune 1000 companies to gauge their interest in buying a contract for yearly inspection service. The results were favorable. Fifty-seven percent of the respondents said that they would consider buying or would definitely buy an inspection contract.

The InstantQA service relies on a combination of technology and in-house software quality expertise to complete the inspection procedure. The InstantQA analysis toolset is founded on a software inspection and transformation engine called the Reasoning5 Code-base Management System (CBMS). This is the driving force behind the application source code profile and analysis, which can adapt to recognize diverse coding practices. This tool works in conjunction with a Reasoning software quality expert versed in current inspection technology and software integrity analysis. InstantQA also incorporates Reasoning's Independent Verification and Validation (IV&V) capabilities, and a unique defect-tracking database analysis, to help direct inspection efforts.

Because it is offered as a service instead of a product, InstantQA can be tailored to specific conditions utilizing user input and requirements, providing a pliant inspection environment. Inspection results are stored in a proprietary information repository, which can be employed in automated re-inspections. The service structure combines a flexible inspection environment with the advantages of iterative analysis. This automated technique narrows the focus of any inspection initiative, which helps mitigate specific software quality issues and determine the extent of the inspection required.

Reasoning's InstantQA employs a tiered methodology, and the inspection takes place in three phases: Preliminary Inspection, Custom Inspection, and Scheduled Re-inspection.

The Method Behind the Model

Preliminary Inspection. This first level of the investigation looks at selected source code and measures it against pertinent "defect classes" that may affect the application functionality. This process helps classify possible data irregularities or application impediments and helps document potential problem areas in the application flow. Preliminary inspection discoveries are displayed in report format to provide a detailed view of the findings and code inventory.

The grand design behind these tools and services is to enable a deeper understanding of an application's properties at the code level, and reveal the impact of any changes made to the code itself.

Custom Inspection. Reviewing the defect classifications established in the preliminary inspection, users and Reasoning analysts combine forces to uncover key problem areas, such as defect types and coding uniformity issues. This collaboration culminates in a highly focused automated inspection performed on the user's source code. A series of inspections is completed to clarify irregularities meaningful to the user's business operation. As the analysis narrows the inspection scope, other less vital defect disclosures are disregarded. This stage concludes with the presentation of a Software Reliability Report that summarizes results and exhibits suggestions on how to improve the software by eliminating defectiveness.

Scheduled Re-inspection. The purpose of scheduled re-inspection is to uphold the integrity of the application over time. This feature protects the initial investment and effort put into ensuring application quality to keep applications defect free.

Combined, these three stages help to prioritize inspection requirements, offer a detailed inspection of application properties, and identify key software defects. The iterative analysis model is beneficial in determining specific defect detection criteria.

Instant QA Service Model

Reasoning's InstantQA employs a tiered methodology, and the inspection takes place in three phases: Preliminary Inspection, Custom Inspection, and Scheduled Re-inspection.

Reasoning's InstantQA is an innovative, but still new, service concept. It will appeal to companies that prefer to purchase off-the-shelf packages and tools that may still require a degree of customization. By acting as a third-party code inspection provider, Reasoning intends to streamline the inspection process by devoting its own internal resources to complete the analysis. This ultimately takes the responsibility off the user as well as offers quick inspection turnaround. Pricing is per inspection engagement and is determined by size, length, and the number of repeat inspections.

The Standish Group believes focus and flexibility are crucial in dealing with a wide range of coding practices, languages, and business environments. InstantQA's comprehensive service architecture has both of these attributes, making it a viable solution. Nevertheless, one factor that might hinder Reasoning's expansion is the immaturity of the market. The Standish Group does forecast growth potential, but not an immediate volcanic reaction. Reasoning's inspection service may be an effective method, but it remains largely untested in the market.

Uncertain Times

Likewise, beyond 2000 lies uncertainty. This does not mean that everything will come to a screeching halt, but as we push past 2000, the efforts of the IT community will be put to the ultimate test. Many IT executives are crossing their fingers in hopes that their technology comes out relatively unscathed, but companies cannot afford to wait patiently on the sidelines.

The Standish Group believes that application development projects will continue at an accelerated rate. Testing remains an intrinsic part of the application development lifecycle, but testing alone is not enough. Software inspection complements testing with its potent analysis capabilities.

And yet, acceptance is not universal, and time remains a barrier. In The Standish Group's Demands Requirement Tracking Study, we asked IT executives from Fortune 1000 companies, "What percent of your new/changing applications will be tested by automated testing or inspection tools?" While 44% of the respondents had a positive response, 56% indicated their applications would not be tested using these types of tools. This revelation doesn't bode well for the IT community.

The cornerstones of a valuable inspection service are depth of analysis, range of code coverage, and service flexibility. The Standish Group believes understanding the value of code inspection is vital to assuring the continuing quality of applications.

Suppliers of these services and inspection tools are advancing into a new market space. Reasoning is following suit with a worthy service, but it can't catch everything. It's not foolproof. Then again, no testing or inspection method is. Application bugs continue to exist in any application, no matter the scope of the analysis. Nevertheless, the more bugs that are caught through automated testing and inspection lessen the damage done to vital business applications.

Kyle Connors is a research associate for The Standish Group International Inc.