Skip to content

Test Outcomes of tests that ran within a Test Cycle

← Worksoft SaaS Analytics : Reports & Dashboards


Before you read this you may want to see:

    Main Article

    The 'Test Cycle based Test Outcomes' set of reports will provide rich analytics about the Quality Assurance initiatives in the context of specific Test Cycles.

    Using reports under this category will give you a greater understanding of how quality assurance efforts are operating and to what extent they are actually succeeding within your project(s), using Test Cycles as the key dimension.

    Specifically, these report will help you visualize/understand how the test outcomes vary by application release/iterations (test cycles) across numerous dimensions like project environment, product flavor, browser/operating system platform, etc.,

    These rich analytics can be accessed in various ways across numerous dimensions:

    • At Project Environment
    • At a Test Cycle Level By Run Definition and By Application Module
    • By Browser / Operating System Platforms By Application Module and By Run Definition
    To generate this reports, there are prerequisites that need to be defined and that would help you achieve desired output.

    Navigate to Labels module:

    Step 1: Create Child Label(s) to System Created Label - Application Modules. You may create all the modules that are available within your application as child labels.

    Step 2: Map Test Scripts only to child level modules and not to System Label – Application Module

    Step 3: Map all Test Scripts that are functional to a label - Automation Entity Purpose-Usage Step.  
    Test scripts that are mapped to ‘Usage Step’ label are the only ones that appear in a report.

    Step 4: Map all functional Run Definitions to Automation Entity Purpose-Business Workflow

     We believe individual business work flows are part of individual Run Definition.To view reports based on individual work, you need to map the Run Definition(s) to child label - Business workflow of the system created label - Automation Entity Purpose.

    Step 5: Execute the Test Run with Test Cycle Identifier.In case of any failures map the Test Runs from the   execution screen to System created Label - Root Cause for Failure

     Upon completion of the Test Runs that belong to the Test Cycle Identifier, navigate to the execution screen of the failed test runs. Map the reason for failure in the "Tag Label" drop down and Save. Root Cause for Failure is Worksoft SaaS system created label which would help you fetch reports based on the failure it is mapped to. If the failure reason listed is not applicable or irrelevant to the failure you may create child labels with the relevant reason or most commonly faced failure reason before you map them.

    Step 6: Navigate to Analytics home page and generate Customised Reports based on the Test Cycle    Identifier provided during the execution in the Test Cycle Level Test Outcomes.

    Once the reports are generated based on the Test Cycle Identifier, you can drill down and view the results for the particular environment, By.,
    • Run Definition 
    • Application Module 

    By Run Definition 

    By Run Definition gives the results at Test Cycle Level and also the list of all the Run Definitions that are mapped to the Business Workflow label.
    Drilling down by Test Cycle level will give you information on
    • Test Outcomes by Run Definition - This explains the Number of Test Runs executed for each Run Definition and their status - Pass/Fail
    Note: System Aborted Runs are not shown in this report
    • Test Failures by Root Cause- This explains the failure reason in your tests to help you prioritize the module or plan the necessary steps to be taken to fix the failure. (This is visible only if the Root Cause for failure label is mapped to failed Test Run before generating the Customized Reports.)
    • Test Execution Times - Show the execution time of the Run Definition in seconds. This also shows you the outliers in the executions.
    • Impact on Outcomes by Duration of Test Execution - A very useful report then tells you the span of  Test Runs executed time in Percentage.
    • Percentage of Passed Test Instructions - This reports will help you understand that the failure is not for all the instructions but there are instructions that have pass 100% or at some percentage and the failure might be really small.
     Access each Run Definition from the list of Run Definitions to view the reports
    •  By Browser / Operating System Platforms - This gives you the pass/fail details based different Browsers/OS the Test Runs are scheduled on.
    •  By Application Module - For the specific Run Definition, mapped to different Application Modules, it gives the data on the pass and failure on the module. This helps in drilling down to the module directly once the failure is identified and fix the issue.
    These reports are generated in PDF, these can also be downloaded by clicking on the PDF icon.   

    By Application Module

    By Application Module gives the results at Test Cycle Level and also the list of all the Modules that are mapped to the Application Module label.
    Drilling down by Test Cycle level will give you information on

    • Test Outcomes by Application Module - This explains the Number of Test Runs executed for each application module that is a part of single or multiple Run Definitions and their status - Pass/Fail
    • Test Failures by Root Cause- This explains the failure reason in your tests to help you prioritize the module or plan the necessary steps to be taken to fix the failure. (This is visible only if the Root Cause for failure label is mapped to failed Test Run before generating the Customized Reports.)
    • Test Execution Times - Show the execution time of the Application Module in seconds. This also shows you the outliers in the executions.
    • Impact on Outcomes by Duration of Test Execution - A very useful report then tells you the span of executed time in Percentage by the specific Application Module.
    • Percentage of Passed Test Instructions - This reports will help you understand that the failure is not for all the instructions but there are instructions that have pass 100% or at some percentage and the failure might be really small.
     Access each Application Module from the list available to view the reports
    •  By Browser / Operating System Platforms - This gives you the pass/fail details based different Browsers/OS the specific module(s) is scheduled on.
    •  By Run Definition - For the specific Application Module, mapped to different Run Definitions, it gives the data on the pass and failure on the Run Definition. 
    These reports are generated in PDF, these can also be downloaded by clicking on the PDF icon.     

    After you read this article you may want to see:



      Feedback and Knowledge Base

      (thinking…)