Skip to content

Accessing the "Summary" for a Test Execution

← Running Tests & Viewing Test Execution Results

Reviewing Summary Information


You have scheduled a test execution and would like to now track the status, duration and other details. This section will explain what all details are available and how to access them.



Every test execution in Worksoft SaaS is identified by a unique id. This is a combination of alphanumerics followed by a uniqueId. This comes handy when you have to share specific execution details with your team or if you want to communicate with the Worksoft SaaS support team for details.

The RunId can be found at the top right section of the execution results screen and that can be copied to the clipboard when clicked on the icon beside the RunId.

                                           


Status of the test execution can be seen on the top right of the page and just below the RunId. Test execution in Worksoft SaaS goes through multiple states. It is important that you get yourself apprised of these to understand if your test is completed and if it yielded desired results.

                                                       

If the execution has failed either during preprocessing or while creating a session and/or while the execution is in progress, a bubble icon (i) is shown beside the status text. Doing a mouseover on the icon displays the reason for the failure.

Status - Queued

When a test is scheduled, Worksoft SaaS creates an instance of the Test Script/Scenario/Run Definition and gives it a unique Id (RunId.) Even before creating an instance, few checks are performed in the system for verifying concurrency & capacity available for the Project & the User. Once the checks are successful, the instance is created.

When the instance is created, the Run is queued and that is the status you see.

Status - Preprocessing In-Progress

When a test is ready to be executed or when you explicitly trigger a test run, preprocessing the status of the test run shows as "Preprocessing In-Progress". During this time, the system binds the data, replaces user-defined variables with respective values and performs other validations. While triggering pre-process from QaCONNECT service, it is possible to avoid validations and thereby gain some time. You can read more about QaCONNECT service(s) here.

Status - Preprocessing Failed

It is possible that preprocessing fails because of different reasons. The reasons could be to do with data, data filters, system-related, etc. When the preprocessing fails system reflects the status and you would see "Preprocessing Failed" text in the status.

Status - Preprocessing Completed

If there are no issues with data and the validations performed are successful, then the state of the test run would be moved to Preprocessing Completed and the same reflects in the status field.

Status - Scheduled

If the schedule (the time when the test run is supposed to start) is in future OR the concurrency assigned to the domain/project/user is already being used up, the status continues to show as Preprocessing Completed until it is time for the test to be picked. 

All the tests that are scheduled from Worksoft SaaS UI, will have the status switch from Preprocessing Completed to "Scheduled" state within a couple of seconds if the concurrency is available.

Status - In Progress

A scheduled run is moved to the "In Progress" state once the VM is launched and the browser is started. Depending on the combination of devices selected (testing context), this may take from a couple of seconds to a few minutes. Typically tests scheduled on devices (real or emulators) take a little longer to launch compared to the desktop browsers.

Status - Completed

Once the test run attains logical conclusion and the VM is destroyed, the test execution is updated to the "Completed" state.

Status - Aborted

When a user aborts a test run or the test run execution is aborted for reasons like runtime exceptions like data issues etc. In such cases, the status of the test run shows as "Aborted".

Status - System Aborted

A test execution can be aborted by the system for various reasons (some VM or component issue) and can happen from any state. When execution is system aborted, any minutes consumed for the test execution are credited to the account.

Status - Cancelled

If the test runs haven't started executing and the user decided to abort the test or capacity for the day is consumed, the test is marked as canceled and the same text "Cancelled" is shown in the status field.


While on test run results page with multiple tabs (results page) opened, you may be lost on what test execution you are viewing/reviewing. To provide you a quick context, the name and status of the entity are shown on the top left.

Test Cycle Details (applicable for cloud executions only)

A test execution can be part of a batch of test runs executed and thus be part of a Test Cycle (refer to Test Cycle article here for details). These details can be reviewed by clicking on the icon beside the name. The system indicates if a test run is linked to a test cycle or not by either showing the icon in "greyed out" or in "orange" color.

           

Also, if you wish to link, delink or change the test cycle to the test run, the same can be accomplished from here.


When you do a mouseover on the icon, you are presented with a tool tipster, where you can click on the "Assign" link to open the test cycle popup. Open Test cycles created in the project are shown in the dropdown. On choosing a test cycle, corresponding details like planned execution time etc are shown.

You can link the test to the test cycle by clicking on the Save button.

               


If a test run is linked to a test cycle, doing a mouse over on the icon will bring up test cycle details in a tooltipster. In addition to the test cycle details, an icon is shown that can be used to de-link the test cycle. Clicking on the icon will remove the link between the test run & the test cycle.

               


If a test run is linked to a test cycle, doing a mouse over on the icon will bring up test cycle details in a tooltipster. In addition to these details, an icon for changing the test cycle is presented. Click on the icon to view the list of test cycles in a dropdown. Choose the test cycle you want to the link the test run to and click on Save to complete the operation.

       


The tool tipster presented for showing the test cycle details has limited information about the test cycle i.e. planned start, planned end, etc. But, if you want to view other details like how many tests are performed in the test cycle and corresponding status, you can click on the View Test Cycle icon on the tooltipster. This would land you on the Test Cycle Home Screen with the corresponding test cycle filtered.


Viewing Bind Data (applicable for cloud executions only)


A test run can be scheduled multiple ways i.e. from Worksoft SaaS or by using QaCONNECT services. When a test is scheduled using QaCONNECT services, it allows for different configurations to be overridden (click here to refer to the help article.)

If you want to view what data is being used by the test run and/or what configurations are overridden, click on the icon to open a popup wherein the details are presented. 
Following details are available:
  • Overriding Data files (if specified during scheduling the test)
  • User-defined variables/values (if specified during scheduling the test)
  • Labels (if specified at the time of scheduling the test)
  • Asset Capture Mode


Viewing User Notes (applicable for cloud executions only)


When a test run is scheduled from QaCONNECT service(s), you can specify free text under the User Notes field.

For example, you use the same Run Definition, that is built to perform a product search, for testing different kinds of searches. You will need a mechanism to differentiate these tests after they are executed. In such instances, you can use User Notes to specify for which product search test case the test run is triggered. While the same information can be gleaned by reviewing test results, this provides a quick way of finding information.


This section/pane shows up when a test run completes with one or more failures. This is intended to help you cut down the time it takes to access the first point of failure and help you with debugging and troubleshooting the failure. Icons in the section provides functionality that allows you to quickly check on the first point of failures in the test run. There are four icons available and provide different sets of information or functionality about the first point of failure.
  • View Screenshot (if Screenshot Capture is enabled)
  • View Error Description
  • View First Point of Failure Details (like command, target, value) &
  • View First Failed Instruction in the Test Execution Results



On clicking the first icon the Asset Viewer opens and the screenshot of the AUT at the first failed instruction gets loaded. This is same as navigating to the first failed instruction and then accessing the screenshot.


The second icon in the pane gives you the reason/description of the instruction that failed. By using this you can avoid having to navigate to the first point of failure and accessing the description.


The third icon in the pane gives you access to the details that can help you with troubleshooting the failures. The details include scenario & script names along with command, target, value and data details.


The fourth & last icon in the section allows you to quickly navigate to the first instruction that failed. On clicking the icon, scenarios & scripts will be expanded and system would auto scroll to the instruction that failed and bring into the view port for quick access.

Failure Triage (applicable for cloud executions only)


This section/pane hosts the functionality you would need once you have completed debugging the test run for failures. This section is displayed irrespective of the run result and following functionality is made available
  • Assign Label(s)
  • Create/Link Issues in Jira &
  • Add Annotation(s)



You have completed debugging and troubleshooting the test run and ready to assign the Root Cause for Failure. Or you want to assign a different label to the test run from what was assigned during scheduling of the test.

You can do this bug using Assign Label component available beside the status text. Click on the button to open the assign label popup. By default "Application Modules", "Root Cause for Failure" and "Test Automation State" labels are shown. To view other labels, you need to select "Show all" radio button.

If a label is already assigned, "Currently applied only" radio option is selected and corresponding labels are shown.





For any of the failures in the executed tests, you can make a note(description) in the Annotations field which will help you to understand why a test run has failed either for yourself to review at a later point or share the information with other peers within your project.

  
You can learn more about the Annotations by clicking here.


When there are any failures in the test run because of any changes to the application, you can integrate Jira within Worksoft SaaS and create/link/delink/view the reported issues.  

You can learn more about the below options by clicking here.




When a test run is scheduled, you can choose the configurations such as Device, browser etc. While reviewing test run results you can refer back to these details provided beside the tabs.

By default, the following details are shown when the user opens the page
  • Cloud used for test execution
  • Application (AUT) environment
  • Device Type
  • Operating System and version
  • Browser and version
To view other details, do a mouse over on the three dots shown after the browser icon.

                                        

Viewing results of multiple test runs without navigating away from the Results page


This is applicable when if you have landed on the page after selecting more than one testing context to execute, in the Testing Context page. In such a case you will be able to switch between the runs by using the drop-down icon beside the testing context details.

Click on the drop-down icon and you will see multiple testing contexts. These are the ones you have chosen in the testing context page and selected to execute. To view run details for the other test run, you can select the required one from the drop down and the page refreshes with execution details of the selected test run.

Viewing Test Execution Statistics


You have executed a test and now would like to know how it performed. You would need to find out how many scenarios/test scripts etc failed. This information along with information like when the test started, completed etc can be found in the summary section.

                                                      


At the top left, you can see three bars in the colors of green or red. Each line stands for one entity i.e. the first bar is for Instructions followed by Test Scripts & Scenarios.

Above each bar, counts are shown in x/y format. Here "x" shows the number of Instructions/Test Scripts/Scenarios that passed and the "y" indicates failed Instructions/Test Scripts/Scenarios. On mouse over you can see the percentage of Instructions/Test Scripts/Scenarios passed and failed.

                                                       

When the test run is scheduled, the counts are not shown. They first show up when the test run moves to "Preprocessing Completed" state. As the test run progresses, these counts keep on changing. The total number of instructions/test scripts, that were shown at the start of the test run may not stay constant as conditional loops & dynamic data binding can impact these counts.




Just beside the bars specifying counts, four timings are mentioned
  • Scheduled Time: When the test is scheduled to start
  • Execution Start Time: In actual when the test execution started
  • Execution End Time: When did the execution end
  • Duration: Time it took for the execution to end (Execution End Time - Execution Start Time)
When the test run is scheduled, only "Scheduled Time" is shown. And the same continues until the test run moves to "In Progress" state. Once the test run moves to In Progress state, Execution Start Time & Duration are populated. While the test is in progress, you can see the duration increase by an interval of 5 seconds. Execution End Time will be populated once the test run completes.

                                                      

Viewing Test Execution Logs (applicable for cloud executions only)


Worksoft SaaS provides different logs to debug and troubleshoot the execution. These logs are available under the "View Logs" section. By default, you see two icons, one for screenshots and the other for video. Others can be found by doing a mouse over on the three dots seen after the video icon.




Following are the logs that you can configure to capture while creating a testing context. These can be overridden while scheduling a test run through QaCONNECT. If the options are chosen, you can view/download these logs from the view logs section.
  • Screenshots (available for all clouds in Worksoft SaaS)
  • Video (available for Worksoft SaaS Cloud & Worksoft SaaS Realtime Cloud)
  • Selenium-Webdriver/appium Logs (available for Worksoft SaaS Cloud & Realtime Cloud)
  • Cloud Log (available for Worksoft SaaS Cloud)
  • Network Log (available for Worksoft SaaS Cloud & High Volumn Cloud)
  • Console Log (available for Worksoft SaaS Cloud)
  • Live Streaming (available for High Volume Testing Cloud)

Except for Screenshots & Live Streaming, the other logs are available once the test run completes. If realtime option is chosen Screenshots can be viewed when the test run is in progress. Live Streaming is available only when the test run is in progress.

To view the logs, you need to click on the specific log you want to view. This will open the "Asset Viewer" and populate the log you have chosen to view. Other available logs are shown in the right pane and you can switch to the log you would like to view.



If none of the logs are chosen while creating the testing context and not overridden when the test is scheduled through QaCONNECT, all the icons would show up in grey and are not clickable.

Asset Viewer (applicable for cloud executions only)


Asset Viewer is an overlay on the execution results screen and this is a container in which you can view all the logs and is divided into four panes as below:
  • Header pane
  • View pane
  • List pane 
  • Metadata pane

This pane shows at the top of the popup and applicable while viewing Screenshots & Video. For other logs, no information is shown in this pane. While debugging a test, you would like to understand the context at that point and given that the popup overlays the results the important information is played in this pane.



While playing the video or viewing a screenshot, following information is displayed
  • Name of the Test Script that is being executed at that instance
  • Command that is being executed at that instance
  • Target that is being executed at that instance &
  • Value that is being executed at that instance

This is the place where the screenshot and other logs are displayed. Depending on the log you selected to view, corresponding information is shown in the pane. When you click on the screenshot icon in the view logs section, screenshot for the first instruction is loaded to the pane.

While viewing the screenshots, you may like to jump to different test scripts. Below the screenshot, you will find a dropdown with previous & next icons beside that.



You can use a dropdown to navigate between different test scripts in the test run. This you may require to jump to a specific test script while debugging an issue

You can also navigate one screenshot at a time (within in the test script) or can move to the first or last of the screenshots by using the buttons available beside the drop down.




This pane can have one or multiple sections depending on the options chosen in the testing context. For example, if you have chosen Yes for Screenshots, Video & Network log, you will see three sections. The corresponding section to the icon you clicked will be default open in the expanded state.

Screenshot section: Thumbnail images of all screenshots pertaining to the test script selected in the dropdown are loaded. You can click on each thumbnail to populate the image in the view pane or use the autoplay option to view all screenshots. The section also displays the total number of screenshots available for the chosen test script. You can find this in the top right corner of the section.

Video view section: Similar to other log sections, this section upon expanding will show one or more thumbnails. If you didn't use "resetSession" command anywhere in the test, there would be only one thumbnail with start & end command instructions showing below the thumbnail. If a "resetSession" command is used in the test, there would be two video thumbnails shown as a new session is established after resetSession.

The other logs follow the same principle of the video view section. When a specific log is chosen it plays in the view pane.

In addition to viewing the logs, they can be downloaded as well. Either you can choose an individual thumbnail to download the corresponding log or you can select all and download. The time it takes to download will depend on the size and the number of logs selected for downloading.



For non-video logs that are of a size greater than 5 MB, only download functionality is made available. They won't open up in the view pane.


This pane displays additional information that may be of use while debugging a test execution. It includes Test Scenario, Test Script, Instruction Sequence number, Loop Iteration, and Data file details. Except for screenshots the last three are not shown.


Download Results (applicable for cloud executions only)


You may come across a requirement where you have to share specific test run results with other teams in your company or other team members who don't have a user account in Worksoft SaaS. You can do that by either downloading the excel report or PDF document with or without screenshots embedded in the document.

In the "Download Results" section seen just below the "View Logs" section, there are two icons placed, one for downloading the excel report and the other for PDF reports. There are different options available while downloading the report and you can choose the ones that are applicable to your requirement.



These two reports are available only after a test run is completed. 

Feedback and Knowledge Base

(thinking…)