Here are all terms you may encounter when reading test run results. The concepts are arranged in the order you may encounter them in the Runs and results tab.
Test results for functional testing is a comparison of the expected and the actual value. It can be either Passed or Failed. For comparative testing, it is a list of differences, Diffs, found for the comparisons you set in test cases within test definitions for which the test was run. See also: How to read test run results.
In functional testing result, the Passed result means that the Expected result of the test run was the same as the Actual result.
In functional testing result, the Fail result means that the Expected result of the test run was different from the Actual result and the implemented changes need to be further investigated.
Diffs are results of comparative testing. They are differences detected by Test Center when comparing your current website configuration with the one you want to implement. They are neither good nor bad, some of them might indicate expected outcomes of the configuration change you are testing. It's up to you to decide whether the diff was expected or unexpected. The expected diff is what you want to achieve by implementing tested configuration changes. An unexpected diff might indicate a configuration error that, if activated on production, could lead to a denial of service or other production issues. Such unexpected diffs should be marked as not accepted and investigated.
For example, if you updated the time-to-live for a file name, then you would expect to find a caching option diff in the results for that file name. In that case, you accept this diff to signify that it indicates expected behavior and needs no further investigation.
An accepted diff is a detected difference that was expected by configuration changes.
An unaccepted diff is a detected difference that was unexpected by configuration changes and needs to be further investigated.
It's an additional comparison done by Test Center. If it occurs in the test run results, it means that an embedded object on the site doesn't occur in one of the networks compared during the test run.
The Error occurred message means that the test could not be executed because of an error. To learn more about the error, you need to hover over the message on the interface.
Updated 5 months ago