Testing an Defect Analysis

Testing Fundamentals

Process

Testing is a process used to help identify the correctness, completeness and quality of developed computer software.

Testing helps in Verifying and Validating if the Software is working as it is intended to be working. This involves using Static and Dynamic methodologies to Test the application.

Objectives

Testing is a process of executing a program with the intent of finding an error.
A good test case is one that has a high probability of finding an as yet undiscovered error.
A successful test is one that uncovers an as yet undiscovered error

Test Strategy

Common Synonyms for Test Strategy are Test
Approach and Test Architecture

A good Test Strategy is

Specific
Practical
Justified

The Test Strategy provides the overall guidelines from which all future testing decisions are made. A well-crafted Testing Approach allows the rest of the testing process to be defined more effectively.

Fundamentals of Test Strategy

Creating Test Strategy
Defining Test Strategy

Below is the example of a poorly stated (and probably poorly conceived) Test Strategy:

“We will use black box testing, cause-effect graphing, unit testing and white box testing to test this product against its specification.”

Creating Test Strategy

Inputs for this process:

A description of the required hardware and software components, including test tools. This information comes from the test environment, including test tool data.

A description of roles and responsibilities of the resources required for the test and schedule constraints. This information comes from man-hours and schedules.

Testing methodology. This is based on known standards.

Functional and technical requirements of the application. This information comes from requirements, change request, technical and functional design documents.

Requirements that the system can not provide, e.g. system limitations.

Outputs for this process:

An approved and signed off test strategy document, test plan, including test cases.

Defining Test Strategy

A high level Test Strategy should include
Business Risks
Testing Milestone
Testing Method
Testing Environment

A detailed Test Strategy should cover the following:

Project Scope
Test Objective
Testing Method
Testing Process and Procedures
Test Compliance

Project Scope:

Restate the business objective of the application and define the scope of the testing. The statement should be a list of activities that will be in scope or out of scope. A sample list would include:

* List of software to be tested
* Software configurations to be tested
* Documentation to be validated
* Hardware to be tested

Test Objectives:

The system under test should be measured by its compliance to the requirements and the user acceptance criteria.

Every feature and function must be listed for test inclusion or exclusion, along with a description of the exceptions. The list should be grouped by functional area to add clarity. The following is a basic list of functional areas:

Backup and recovery
Workflow
Interface design
Installation
Procedures (users, operational, installation)
Requirements and design
Messaging
Notifications
Error handling
System exception and third-party application faults

Testing Methods:

The methodology provides the detail necessary to describe the levels and types of testing and should identify types of testing that are needed to validate the system.

Some of the more specific test types include

Static Testing
Dynamic Testing
Black Box Testing
White Box Testing
Unit Testing
Requirement Testing
Regression Testing
Error Handling Testing
Intersystem Testing
Parallel Testing
Stress Testing
Volume Testing
Performance Testing
Manual Testing

Testing Process and Procedures:

The order of test execution and the steps necessary to perform each type of test should be described in sufficient detail to provide clear input into the creation of test plans and test cases.

Procedures should include how test data is created, managed and loaded.

Test cycles should be planned and scheduled based on system availability and deliverable dates from development.

All application and environmental dependencies should be identified along with the procedures necessary to gain access to all the dependent systems.

Test Compliance:

Every level of testing must have a defined set of entry/exit criteria which is used to validate that all prerequisites for a valid test have been met.

All mainstream software testing methodologies provide an extensive list of entry/exit criteria and checklist.

In addition to the standard list, additional items should be added based on specific testing needs. Some common additions are, environmental availability, data availability, validated code which is ready to be tested.

Test Plan

A test plan is a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of what the eventual workflow will be.

In software testing, a test plan gives detailed testing information regarding an upcoming testing effort, including

Scope of testing.
Schedule.
Test Deliverables.
Release Criteria.
Risks and Contingencies

Test Methods

In order to decide on the Test Methods needed for the project/product during Test Strategy Analysis, It is very important to understand various Test Methods and how it works.

Let’s go through some of the Industry Standard Test methods which are being used.

Static Testing

The Verification activities fall into the category of Static Testing. During static testing, you have a checklist to check whether the work you are doing is going as per the set standards of the organization. These standards can be for Coding, Integrating and Deployment. Review’s, Inspection’s and Walkthrough’s are static testing.

Static testing involves review of requirements or specifications. This is done with an eye toward completeness or appropriateness for the task at hand. This is the verification portion of Verification and Validation.

Dynamic Testing

Dynamic Testing involves working with the software, giving input values and checking if the output is as expected. These are the Validation activities.

Unit Tests, Integration Tests, System Tests and Acceptance Tests are few of the Dynamic Testing methodologies.

In dynamic testing the software must actually be compiled and run; this is in contrast to static testing. Dynamic testing is the validation portion of Verification and Validation.

Black Box Testing

Black Box Testing is not a type of testing; it instead is a testing strategy, which does not need any knowledge of internal design or code etc.

As the name “black box” suggests, no knowledge of internal logic or code structure is required. The types of testing under this strategy are totally based/focused on the testing for requirements and functionality of the work product/software application.

Black box testing is sometimes also called as “Opaque Testing”, “Functional/Behavioral Testing“ and “Closed Box Testing”

Black Box Testing Approach

Testing methods where user is not required

Functional Testing:

In this type of testing, the software is tested for the functional requirements. The tests are written in order to check if the application behaves as expected.

Load Testing:

The application is tested against heavy loads or inputs such as testing of web sites in order to find out at what point the web-site/application fails or at what point its performance degrades.

Ad-hoc Testing:

This type of testing is done without any formal Test Plan or Test Case creation. Ad-hoc testing helps in deciding the scope and duration of the various other testing and it also helps testers in learning the application prior starting with any other testing.

Exploratory Testing:

This testing is similar to the ad-hoc testing and is done in order to learn/explore the application.

Usability Testing:

This testing is also called as ‘Testing for User-Friendliness’. This testing is done if User Interface of the application stands an important consideration and needs to be specific for the specific type of user.

Smoke Testing:

This type of testing is also called sanity testing and is done in order to check if the application is ready for further major testing and is working properly without failing up to least expected level.

Recovery Testing:

Recovery testing is basically done in order to check how fast and better the application can recover against any type of crash or hardware failure etc. Type or extent of recovery is specified in the requirement specifications.

In this type of testing, the software is handed over to the user in order to find out if the software meets the user expectations and works as it is expected to.

Alpha Testing:

In this type of testing, the users are invited at the development center where they use the application and the developers note every particular input or action carried out by the user. Any type of abnormal behavior of the system is noted and rectified by the developers.

Beta Testing:

In this type of testing, the software is distributed as a beta version to the users and users test the application at their sites. As the users explore the software, in case if any exception/defect occurs that is reported to the developers.

Advantages of Black Box Testing

More effective on larger units of code than glass box testing
Tester needs no knowledge of implementation, including specific programming languages
Tester and programmer are independent of each other
Tests are done from a user’s point of view
Will help to expose any ambiguities or inconsistencies in the specifications
Test cases can be designed as soon as the specifications are complete

White Box Testing

White box testing can be performed to validate whether code implementation follows intended design, to validate implemented security functionality, and to uncover exploitable Vulnerabilities

White box testing includes analyzing data flow, control flow, information flow, coding practices, and exception and error handling within the system, to test the intended and unintended software behavior.

White box testing requires knowing what makes software secure or insecure, how to think like an attacker, and how to use different testing tools and techniques.

Techniques

Data Flow Analysis
Code-Based Fault Injection
Abuse Cases

Checkpoints

Data
Component Interfaces
Configuration
Error Handling

White Box Testing – Results to Expect

The types of errors uncovered during white box testing are several and are very context sensitive to the software under test.
Some examples of errors uncovered include

data inputs compromising security
sensitive data being exposed to unauthorized users
improper control flows compromising security
incorrect implementations of security functionality
unintended software behavior that has security implications
design flaws not apparent from the design specification
boundary limitations not apparent at the interface level
White box testing greatly enhances overall test effectiveness and test coverage. It can greatly improve productivity in uncovering bugs that are hard to find with black box testing or other testing methods alone.

Pros and Cons of White Box Testing

Pros
White box testing uncovers programming and implementation errors.
Finding ”unintended” features can be quicker during white box testing.
White box testing is typically very effective in validating design decisions and assumptions from the security context.

Cons
White box testing is time consuming and expensive and requires specialized skills.

Unit Testing
Unit testing is a procedure used to validate that individual units of source code are working properly.

A unit may be an individual program, function, procedure, base/super class, abstract class or derived/child class.
The goal of unit testing is to isolate each part of the program and show that the individual parts are correct.
It simplifies integrations. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier.

Limitations of Unit Testing

By definition, it only tests the functionality of the units themselves. Therefore, it may not catch integration errors, performance problems, or other system-wide issues.

Unit testing is more effective if it is used in conjunction with other software testing activities.

Requirement Testing

Requirement Testing starts at the beginning of the project, not at the end of the coding.
It apply tests to assure the quality of the requirements. Then the later stages of the project can concentrate on testing for good design and good code.

To pass through the quality gateway and be included in the requirements specification, a requirement must pass a number of tests.

Quality Measurement
Quantifiable Requirement
Non Quantifiable Requirement
Coherency and Consistency
Completeness
Relevance
Requirement or Solution
Stakeholder Value
Traceability
Ordering / Grouping

Advantages of Requirement Testing
The key advantage of this approach is that we minimize expensive rework by minimizing requirements-related defects that could have been discovered, or prevented, early in the project’s life.

Regression Testing

Regression testing is any type of software testing which seeks to uncover regression bugs.

Typically regression bugs occur as an unintended consequence of program changes.

One of the common methods of regression testing is re-running previously run tests and checking whether previously fixed faults have re-emerged.

During development, this kind of re-emergence of faults is common. Sometimes it occurs because a fix gets lost through poor revision control practices or simple human error in revision control.

Strategies and Factors
Some strategies and factors to consider during regression testing include the following:

Test fixed bugs promptly.
Watch for side effects of fixes. The bug itself might be fixed but the fix might create other bugs.
If two or more tests are similar, determine which is less effective and get rid of it.
Identify tests that the program consistently passes and archive them.
Focus on functional issues, not those related to design.
Make changes (small and large) to data and find any resulting corruption.

Error-Handling Testing

Usage
It determines the ability of applications system to process the incorrect transactions properly
Errors encompass all unexpected conditions.

Objective
Determine Application system recognizes all expected error conditions.

How to Use

A group of knowledgeable people is required to anticipate what can go wrong in the application system.
Then logical test error conditions should be created based on this assimilated information.

When to Use

Throughout SDLC

Examples

Create a set of erroneous transactions and enter them into the application system then find out whether the system is able to identify the problems.

Using iterative testing enter transactions and trap errors. Correct them. Then enter transactions with errors, which were not present in the system earlier.

Inter-System Testing

Objective

Determine Proper parameters and data are correctly passed between the applications.
Documentation for involved system is correct and accurate.
Ensure Proper timing and coordination of functions exists between the application system.

How to Use
Operations of multiple systems are tested.
Multiple systems are run from one another to check that they are acceptable and processed properly.

When to Use
When there is change in parameters in application system.
The parameters, which are erroneous then risk associated to such parameters, would decide the extent of testing and type of testing.

Example
Develop test transaction set in one application and passing to another system to verify the processing.

Parallel Testing

Parallel/audit testing is a type of testing where the tester reconciles the output of the new system to the output of the current system, in order to verify the new system operates correctly.

Parallel testing is common in many HRMS applications

When to Use

Parallel testing is not so much about finding problems, but acts as a “dry run” for production.
How to Use
The new system is tested alongside the legacy system it is replacing with all the same variables.
The validation compares the output of the legacy system to the output of the new system and the results should match.
Who to Use
People running a parallel test are the same people that will
be responsible for the system once it is in production.
QA is still involved, but their role is more in the process than in the execution.

Stress Testing

Stress testing deals with the quality of the application in the environment. The idea is to create an environment more demanding of the application than the application would experience under normal work loads. This is the hardest and most complex category of testing to accomplish and it requires a joint effort from all teams.

Objective

The primary objective of the stress testing is to ensure the software doesn’t crash in conditions of
Insufficient computational resources (such as memory or disk space),
Unusually high concurrency, or denial of service attacks.
Example

A web server may be stress tested using scripts, bots, and various denial of service tools to observe the performance of a web site during peak loads.

Volume Testing

Volume testing refers to testing a software application for a certain data volume. This volume can in generic terms be the database size or it could also be the size of an interface file that is the subject of volume testing.

Volume testing comes under non-functional requirements testing and Performance testing.

Volume testing will also be undertaken (normally) as part of the User Acceptance test. Stress testing is closely related, as it seeks to find out how the software will behave beyond its specified limits.

Why to Use

To identify where the behavior of the software deviates from that expected for a specified volume of data.

Who to Use

Developers through to customers and end-users can do it.

How to Use

Volume testing needs two things.
Firstly clear expected outcomes of how the software is to behave for a given level of data.
Secondly, data, and lots of it.

Performance Testing
System performance is generally assessed in terms of response time and throughput rates under differing processing and configuration conditions.
Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device.
Performance testing can verify that a system meets the specifications claimed by its manufacturer or vendor.

Benefits

Removal of serious functional flaws related to data volume and synchronization issues not identified by functional testing.

Collection of performance metrics for the system under increasing load that enable system administrators to make and validate tuning decisions prior to deployment.

Using realistic data volumes and transaction rates enables identification of processing bottlenecks.

Manual Testing

Manual testing is the oldest and most rigorous type of software testing. Manual testing requires a tester to perform manual test operations on the test software without the help of Test automation.

Q: Can Manual Testing be directly replaced by automated testing?

A: Manual testing can be replaced by test automation. It is possible to record and playback manual steps and write automated test script(s) using Test automation tools. Although, test automation tools will only help execute test scripts written primarily for executing a particular specification and functionality, it lacks the ability of decision-making and recording any unscripted discrepancies during program execution.

Defect Analysis
Defect Analysis is nothing but the data on number of defects in a typical system, percentage of time spent on testing, rate of finding defects, etc.

Three essential components are to be in place in order to do a proper Defect Analysis.

Defect Detection
Defect Reporting
Defect Abstracts

Defect Detection

The typical defect detection process involves

Identifying the defect
Recording the defect details
Recording the defect type
Recording the defect priority
Setting the status for defect
Fixing the defect
Resetting the status of defect

Defect Reporting

Defect reports are among the most important deliverables to come out of test.
Effective defect reports will:
Reduce the number of defects returned from development.
Improve the speed of getting defect fixes.
Improve the credibility of test
Enhance teamwork between test and development
Why do some testers get a much better response from development than others? Part of the answer lies in the defect report.

Here are some key points to make sure the next defect report you write is an effective one.

1. Condense – Say it clearly but briefly
2. Accurate – Is it a defect or could it be user error, misunderstanding, etc.?
3. Neutralize – Just the facts. No zingers. No humor. No emotion.
4. Precise – Explicitly, what is the problem?
5. Isolate – What has been done to isolate the problem?
6. Generalize – What has been done to understand how general the problem is?
7. Re-create – What are the essentials in triggering/re-creating this problem?

Environment, steps, conditions
8. Impact – What is the impact to the customer? What is the impact to test?
Sell the defect.
9. Debug – What does development need to make it easier to debug?
Traces, dumps, logs, immediate access, etc.
10. Evidence – What will prove the existence of the error?

Defect Abstracts
The short one line abstract that gets associated with most defects is a very powerful communication tool. Often times, the abstract is the only portion of the defect that gets read by the decision-makers.

Example 1: The following abstract is true but doesn’t provide nearly as much information as it could.
Abstract 1: Problems found when saving and restoring local data.
Example 2: Perhaps a more descriptive abstract would be.
Abstract 2: Save/Restore of Local data on Quotation fails.