Key Strategies to Conduct Regression Testing for Application Modernization

Mobile applications are constantly evolving and updating to meet the needs and expectations of their users. However, every change in the code or functionality of the app can introduce new bugs or break existing features. A recent study said that over 44% of defects are introduced when new features are added or existing ones are modified. This is why mobile regression testing is essential, along with the best strategies to get the most out of your regression tests.

What is mobile app regression testing?

Mobile regression testing is the process of verifying that the existing features and functionalities of a mobile application work as expected after changes or updates are made to the app. It involves re-running the test cases that cover the core features of the app, checking for any defects or errors, and reporting them.

Why do we need mobile regression testing?

Mobile regression, or regression testing in general, is really important for any application and web. Here’s why:

• Reduces the risk of releasing a faulty app to the market and facing negative reviews.
• Saves time and money by detecting and fixing bugs early in the development cycle.
• Ensures compatibility of the app with different mobile devices, operating systems, and network conditions.
• Maintains the quality of the mobile application throughout the app lifecycle.

Main challenges of mobile regression testing

Before going to the best strategies for approaching mobile regression testing, let’s understand what types of challenges a tester faces whilst performing regression for mobile applications.

• Different devices and fragmentation of the mobile market require testing the app on various devices, screen sizes, resolutions, orientations, and platforms.
  
• Frequent updates and fast-paced releases of the app, which require testing the app in short and agile cycles, give testers a limited testing window.
  
• Balancing test coverage with development timelines and resource constraints can be challenging with the constant introduction of new modules within a short period of time.
  
• Human errors and oversights are likely to be reproduced due to exhaustive, repetitive testing of the tasks again and again.

Best strategies and approaches for mobile regression

Now let’s get down to the main section of the blog. We will cover the best strategies and approaches for mobile regression, including a description of the strategy and its effect.

Sequential execution

This strategy involves testing the app on multiple devices in a sequence using a cron job that triggers the test execution at a specified time and frequency with the help of a GitHub Actions workflow file.

A yaml file is used to define the device configurations, test cases, and results.

An example of a sequential YAML file:

name: Mobile Regression Tests
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: |
          npm install -g appium
          npm install -g appium-doctor

      # Loop through each device configuration and execute tests
      - name: Execute tests on multiple devices
        run: |
          for device in "iPhone 12" "Samsung Galaxy S20" "Google Pixel 5"; do
            echo "Running tests on $device"
            npm run test --device="$device"
          done

      - name: Upload test results
        uses: actions/upload-artifact@v3
        with:
          name: allureReport
          path: ./allure-report

The effect of this strategy:

With this approach, you will be able to execute the regression tests via cron job upon command on multiple devices sequentially. Remember, this strategy should not be used if your testing time is longer; for that, refer to the second strategy.

Parallel execution

Parallel execution involves running tests simultaneously on multiple devices to reduce overall test execution time. By distributing tests across multiple devices, you can achieve faster feedback and improve the efficiency of regression testing for mobile applications.

An example of a parallel YAML file:

# Execute tests in parallel for each device
      - name: Execute tests on ${{ matrix.device }}
        run: npm run test --device="${{ matrix.device }}"

      - name: Upload test results for ${{ matrix.device }}
        uses: actions/upload-artifact@v3
        with:
          name: allureReport-${{ matrix.device }}
          path: ./allure-report-${{ matrix.device }}

Explanation:

• This workflow defines a job named `test` that runs on the `ubuntu-latest` environment.
  
• The `strategy.matrix.device` directive allows us to define multiple parallel executions, one for each device specified in the matrix.
  
• The steps within the job are executed for each device in parallel.
  
• Each step executes tests on a specific device using the `npm run test --device="${{ matrix.device }}"` command.
  
• Test results for each device are uploaded as artifacts with a unique name (`allureReport-${{ matrix.device }}`).

Report sharing integration

The next strategy is, after your daily regression automation gets completed, to provide an overview of results every day to your stakeholders, which becomes a bit hectic. Instead,  integrate report sharing with communication channels like Slack.

How to do it?

Follow the steps to integrate slack reporting.

1. Set up a Slack Incoming Webhook ‍
   • Go to your Slack workspace and navigate to the channel or direct message where you want to receive the test reports.
   • Click on "Add an app" or the "+" icon and search for "Incoming Webhooks."
   • Click on "Add to Slack" and select the channel where you want to receive notifications.
   • Follow the prompts to configure the webhook, and note down the webhook URL generated by Slack.
2. Install the required dependencies
   Install the Axios library for making HTTP requests to the Slack webhook.
   npm install axios
3. Set up environment variables
   Create a .env file in your project directory to store sensitive information, such as Slack webhook URLs. Add the Slack webhook URL as an environment variable in the .env file:
   SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
4. Import the required modules
   Import the necessary modules into your test automation script. In this example, we import Axios for making HTTP requests and the dotenv package for reading environment variables from the .env file.
   const axios = require('axios').default;
   require('dotenv').config();
5. Define the reporting function
   Define a function to generate and send the test execution report to Slack. This function will construct the message payload and make a POST request to the Slack webhook URL.
   ‍
   const reportToSlack = async () => {
    // Construct message payload
    const slackData = {
         "blocks": [
                {
                    "text": {
                                  "text": "Test execution report", // Customize message text
                                  "type": "mrkdwn"
                    },
                    "type": "section"
                }
          ]
     };
     // Make POST request to Slack webhook URL
     await axios.post(process.env.SLACK_WEBHOOK_URL, slackData);
};
   ‍
6. Trigger reporting function
   Call the reportToSlack function at the appropriate point in your test automation script to send the report to Slack.
   
(async () => {
      await reportToSlack();
})();

Test data management

Effective test data management is crucial for mobile regression testing, especially in scenarios where applications rely on dynamic or external data sources. This strategy involves carefully managing test data to achieve good coverage of test scenarios and minimize test maintenance.

Key components for test data management

1. Data generation: Develop automated processes for generating test data sets that cover a wide range of test scenarios, including boundary conditions, edge cases, and real-world usage patterns.
   
2. Data variation: Ensure test data variation by incorporating different data combinations, including valid, invalid, and borderline data inputs.
   
3. Data masking: Implement data masking techniques to protect sensitive information and comply with privacy regulations.

Integrate BDD principles

Implement Behavior-Driven Development (BDD) principles and practices to drive the creation of regression test scenarios that focus on desired application behaviors and user interactions. BDD promotes collaboration between stakeholders, developers, and testers by using a common language to define test requirements and acceptance criteria.

Key points for BDD

‍Use of Gherkin syntax:

Write test scenarios using Gherkin syntax, a structured language that uses plain-text representations to describe application behavior. Gherkin features keywords such as Given, When, and Then, which facilitate clear and concise communication of test intent.

Clear and Descriptive titles: 

Use descriptive and meaningful scenario titles that accurately reflect the intended behavior being tested. Clear scenario titles make it easier for team members to understand the purpose of each test.

Feature: Shopping cart functionality
  Scenario: Add item to cart
    Given the user is on the item detail page
    When the user adds an item to the cart
    Then the item should be visible in the cart

Use of Given-When-Then structure:

Structure test scenarios according to the Given-When-Then (GWT) pattern, which helps organize tests into logical steps representing the initial context, action, and expected outcome. This structured approach improves the readability of test scenarios.

Naming conventions for steps:

Adopt consistent naming conventions for the Given, When, and Then steps to ensure clarity and consistency across test scenarios. Use descriptive and meaningful names that convey the intent of each step and promote understanding among team members.

Scenario comments:

Include comments within test scenarios to provide additional context, explanations, or references to related requirements or documentation.

Feature: Payment process
  Scenario: Complete checkout process
    # This scenario tests the end-to-end checkout process
    Given the user has items in the shopping cart
    When the user proceeds to checkout
    Then the payment should be processed successfully

Conclusion

In summary, the reason we perform mobile app regression testing is to maintain the app’s quality throughout its lifecycle, and these strategies are just to ease the regression process and also boost the effectiveness of your regression test suite.

If you still have any more queries regarding regression testing, mobile testing, or testing in general, reach out to us at Alphabin.