System Testing is one of the most difficult activities and is also mostly misunderstood. It is the process of human testing. System testing is conducted on a complete, integrated system to evaluate the system’s compliance with its specified requirements. It verifies software operation from the perspective of the end-user, with different configuration setups. System testing falls within the scope of Black box testing. On building the entire system, it needs to be tested against the “System Specification” to check whether it delivers the required features. The System Testing begins with software components that have cleared Unit and Integration Testing. System Testing can be conducted in parallel with function Testing.
The steps of system testing are:
- Decompose and analyze the SRS (Software Requirements Specification).
- Partition the requirements.
- Determine inputs and outputs for requirements.
- Develop the requirements test cases.
- Develop a requirements coverage matrix, which is simply a table that maps the requirements with the test cases.
- Execute the test cases and measure logic coverage.
- Develop additional tests, as indicated by the combined coverage information. Various special tests are evolved to check whether functional and non-functional requirements of the software have been met.
System Testing Types:
- Usability testing
- Performance Testing
- Load Testing
- Stress Testing
- Security Testing
- Configuration testing
- Compatibility testing
- Installability testing
- Recovery Testing
- Availability testing
- Volume testing
- Accessibility testing
- Localization Testing
- Internationalization Testing
All these tests are conducted for non-functional requirements, whereas Security Testing is conducted for both functional and non-functional requirements.
Usability Testing:
System usability testing is the system testing of an integrated, black box application against its usability requirements. The typical aim of usability testing is to cause the application to fail to meet its usability requirements so that the underlying defects can be identified, analyzed, fixed, and prevented in the future.
The system usability test is conducted to:
- Partially validate the application by determining if it fulfills its quantitative and qualitative usability requirements concerning:
- Installation by the environments team.
- Usage by the user organization.
- Operations by the operations organization.
- Identify usability defects involving the application’s human interface, such as:
- Difficulty of orientation and navigation (e.g., navigation defects such as broken links and anchors within a website).
- Efficiency of interactions (based on user task analysis).
- Information consistency and presentation.
- Appropriate use of language and metaphors.
- Conformance to the Digital Brand Description Document Website Design Guidelines.
- Programming defects (e.g., incorrectly functioning tab key, accelerator keys, and mouse actions).
- Determine if the application’s human interfaces should be iterated to make them more usable.
- Help determine the extent to which the application is ready for launch.
- Provide input to the defect trend analysis effort.
- Determine how well the human interfaces enable the different categories of users to efficiently perform their business tasks.
- Report these failures to the development teams so that the associated defects can be fixed.
What is Usability?
Usability is a combination of factors that influence a user’s experience with a product or system. Usability criteria include:
- Ease of learning: How fast can a user learn to use the system that they haven’t encountered before, in order to accomplish basic tasks?
- Efficiency of use: Once a user is experienced with the system, how much time does it require to accomplish key tasks?
- Memorability: When the user approaches the system the next time, do they remember enough in order to use it effectively?
- Error frequency and severity: How often do errors occur in the system? How severe are they? How.
- Error frequency and severity: How frequently do errors occur in the system? How severe are they? How do users recover from errors?
- • Subjective satisfaction: How does the user react to the system? How do they feel about using it?
What is Usability Testing?
Usability testing is a methodical evaluation of the graphical user interface (GUI) according to usability criteria.
Usability testing is the measure of the quality of a user’s experience when interacting with a system, such as a website, software application, mobile technology, or any user-operated device.
What is the purpose of Usability Testing?
The purpose of usability testing is to determine how well target users can understand and use a software or hardware product. It helps establish the ease of use and effectiveness of a product, identifies usability problems, and provides solutions to improve user experience. Implementing these solutions can make the product easier to use, reduce the need for support, and increase its acceptance in the marketplace.
Methods of Usability Testing:
- Laboratory experiments: Testing conducted in a controlled environment where users perform key tasks based on predefined scenarios. User activities are recorded using cameras, and usability experts take notes.
- On-site observation: Similar to laboratory experiments but conducted on-site in a real environment. This method allows for a longer observation period and provides insights from users’ actual work environments.
Summary of Usability Testing:
Usability testing aims to identify discrepancies between a product’s user interface and the requirements of its users. It involves observing users interacting with the product and adapting the software to match their work styles. Usability testing should be performed early in the development cycle and can be repeated multiple times to ensure the application is easy to use and understand.
Performance Testing:
Performance testing is the system testing of an integrated, black box, partial application against its performance requirements under normal operating circumstances. Its purpose is to identify inefficiencies, bottlenecks, and defects related to application performance.
The goals of performance testing include:
- Validating the system’s performance requirements.
- Reporting failures and defects.
- Assisting in performance tuning.
- Reducing hardware costs.
- Determining the system’s readiness for launch.
- Providing input for defect trend analysis.
Performance testing measures response times, throughput, latency, and capacity to ensure the application meets the required performance standards.
Load Testing:
Load testing is the process of subjecting a system under test to statistically representative load conditions to identify its limitations. It helps determine if the application can handle heavy loads and stress scenarios.
The purposes of load testing include:
- Validating scalability requirements.
- Locating performance bottlenecks.
- Reporting failures and defects.
- Determining launch readiness.
- Providing input for defect trend analysis.
Load testing involves running regression tests at maximum loads to identify how the system responds and if it meets performance requirements.
Stress Testing:
Stress testing aims to break the system under test by overwhelming its resources. It helps determine the system’s failure points and evaluates its recoverability.
The purposes of stress testing include:
- Validating scalability requirements.
- Assessing system degradation and failure under extreme conditions.
- Reporting failures and defects.
- Determining launch readiness.
- Providing input for defect trend analysis.
Stress testing induces chaos and unpredictability to ensure the system fails and recovers gracefully.
Security Testing
Security testing is checking a system, application, or its components against its security requirements and the implementation of its security mechanisms. The aim of security testing is to cause the system, application, or component to fail. It also verifies the application’s failure to meet security-related requirements (black box testing), failure to properly implement security mechanisms (white box and gray box testing), thereby enabling the identification, analysis, fixing, and prevention of underlying defects in the future.
The security testing covers:
Requirements: Partially verify the application (i.e., to determine if it fulfills its security requirements):
- Identification
- Authentication
- Authorization
- Content Protection
- Integrity
- Intrusion Detection
- Privacy
System Maintenance Mechanisms: Determine if the system causes any failures concerning its implementation of security mechanisms:
- Encryption and Decryption
- Firewalls
- Personnel Security: Passwords
- Digital Signatures
Cause Failures:
- Cause failures concerning the security requirements that help identify defects that are not efficiently found during other types of testing.
- The application fails to identify and authenticate users.
- The application allows a user to perform an unauthorized function.
- The application fails to protect its content against unauthorized usage.
- The application allows the integrity of data or messages to be violated.
- The application allows undetected intrusion.
- The application fails to ensure privacy by using an inadequate encryption technique.
Report Failures:
- Report these failures to the development teams so that the associated defects can be fixed.
Determine Launch Readiness:
- Help determine the extent to which the system is ready for launch.
- Project Metrics: Help provide project status metrics.
- Trend Analysis: Provide input to the defect trend analysis effort.
What is Security Testing? Security testing is performed to ensure that only users with the appropriate authority can use the applicable features of the system.
What is the purpose of Security Testing? The purpose of security testing is to determine how well a system protects against unauthorized internal or external access or willful damage.
Summary of security testing: Security testing aims to show that the program’s security requirements can be subverted. It tests whether the system meets its specified security objectives. The tester’s aim is to demonstrate the system’s failure to fulfill the stated security requirements. However, it is impossible to prove that a system is impenetrable. The objective is to establish sufficient confidence in security.
Configuration Testing
Configuration testing is done to check whether the system can work on differently configured machines, i.e., software with hardware. Computers are designed using different peripherals, components, drivers, which are designed by various companies.
What is Configuration Testing? Configuration testing is the process of checking the operation of the software you are testing with all these various types of hardware.
What is the purpose of Configuration Testing? The purpose of Configuration Testing is to determine whether the program operates properly when the hardware or software is configured in a required manner.
Summary of Configuration Testing: It is the process of checking the operation of the software with all various types of hardware.
Example: For applications to run on Windows-based PCs used in homes and businesses.
- The PCs: Different manufacturers such as Compaq, Dell, Hewlett Packard, IBM, and others.
- Components: Disk drives, video, sound, modem, and network cards.
- Options and memory.
- Device drivers.
Compatibility Testing
Software compatibility testing means checking that your software interacts with and shares information correctly with other software. For example, it measures how web pages display well on different browser versions.
What is Compatibility Testing? Compatibility testing is used to determine if your software application has issues related to how it functions in connection with the operating system and different types of system hardware and software.
What is the purpose of Compatibility Testing? The purpose of compatibility testing is to evaluate how well software performs in a particular hardware, software, operating system, browser, or network environment.
Summary of Compatibility Testing: Compatibility testing is the process of determining the ability of two or more systems to exchange information. In a situation where the developed software replaces an already working program, an investigation should be conducted to assess possible compatibility problems between the new software and other programs or systems. It means checking that your software interacts with and shares information correctly with other software.
Examples:
- What other software (operating systems, web browsers, etc.) is your software designed to be compatible with?
- Upgrading to a new database program and having all your existing databases load in.
Installation Testing
An installability requirement is a utility requirement that specifies a required amount of installability, which is a quality factor that is defined as follows:
“Installability is the case with which something can be successfully installed in its production environment(s).”
What is Installation Testing? Installation testing (in software engineering) can simply be defined as any testing that occurs outside of the development environment. Such testing will frequently occur on the computer system in which the software product will eventually be installed.
What is the purpose of Installation Testing? Installation Testing is done to identify the ways in which installation procedures lead to incorrect results. It is done to:
- Ensure that the application or component is easy to install.
- Ensure that time and money are not wasted during the installation process.
- Improve the morale of the engineers that will install the application or component.
- Minimize installation defects.
- Determine whether the installation procedure is documented.
- Determine whether the methodology for migration from the old system to the new system is documented.
Summary of Installation Testing: To identify the ways in which the installation procedures lead to incorrect results. Testing installation procedures is a good way to avoid making a bad impression. Since “Installation Makes the first impression on the end-user,” it is the testing concerned with the installation procedures for the system. Installation options are:
- New
- Upgrade
- Customized/Complete
- Under Normal and Abnormal Conditions
Recovery Testing
Recovery testing is done to determine whether operations can be continued after a disaster or after the system has been lost. It involves reverting to a point where the integrity of the system was known and then reprocessing transactions up to the point of failure. It is used where continuity of operations is essential.
What is the purpose of Recovery Testing? The purpose of Recovery Testing is to verify the system’s ability to recover from varying degrees of failure.
Summary of Recovery Testing: To determine whether the system or program meets its requirements for recovery after a failure.
Availability Testing
Availability testing is conducted to check both the reliability and availability of an application. The application is tested for its reliability so that any failures can be discovered and removed before deploying the system. Most likely scenarios are tested under normal usage conditions to validate that the application provides the expected service. As time permits, more complicated tests can be applied to reveal subtle defects.
What is Availability Testing? Availability testing is running an application for a planned period and collecting failure events with repair times. It also compares the availability percentage to the original service level agreement. Where reliability testing focuses on reducing the number of failures, availability testing primarily measures and minimizes the actual repair time.
The formula for calculating percentage availability: (MTBE: 7(MTBI 1 MITR)) X 100. Notice that as MTTR trends towards zero, the percentage availability trends towards 100%. This becomes the essential focus of availability testing: to reduce and eliminate downtime.
Summary of Availability Testing: Availability testing is designed to challenge a system’s ability to manage the maximum amount of data over a period of time. This type of testing also evaluates a system’s ability to handle overload situations in an orderly fashion. It involves testing where the system is subjected to large volumes of data.
Accessibility Testing
Accessibility testing is the process of ensuring that a web application is accessible to people with disabilities. Accessibility verification can help prevent functionality problems that could occur when people with disabilities try to access the application with adaptive devices such as screen readers, refreshable Braille displays, and alternative input devices. In addition, if a web application is produced for or by a US government agency, accessibility verification is required to prevent violation of a federal law, potential loss of government contracts, and potential costly lawsuits.
What is Accessibility Testing? Accessibility testing is an approach to measuring a product’s ability to be easily customized or modified for the benefit of users with disabilities. Users should be able to change input and output. The goal of Accessibility Testing is to ensure that people with disabilities can access and use the software product as effectively as users without disabilities.
What is the purpose of Accessibility Testing? The purpose of accessibility testing is to pinpoint problems within websites and products that may otherwise prevent users with disabilities from accessing the information they are searching for. It can help determine the following:
- Compliance: How your product complies with legal requirements regarding accessibility.
- Effectiveness: How well users with disabilities can use your product to accomplish basic and complex tasks.
- Usefulness: The likelihood that users with disabilities will want to use your product again because it meets their needs and expectations.
- Satisfaction: How appealing your product is to users with disabilities.
Summary of Accessibility Testing: Accessibility testing determines the degree to which the user interface of an application enables users with common or specified disabilities (e.g., auditory, visual, physical, or cognitive) to perform their specified tasks. It enables users with common disabilities to use the application or component.
Examples of accessibility requirements:
- The application shall be usable by persons with auditory disabilities, colorblindness, visual acuity or blindness, and common screen readers.
- Physical disabilities by enabling them to verbally interact with and use it.
- Users with mild cognitive disabilities.