View build reports in Azure Pipelines

Embed your Test Reporting & Analytics build report in the Azure Pipelines run summary using the BrowserStack Azure DevOps extension.

Azure Pipelines is a continuous integration tool that builds, tests, and deploys your code on every change. The BrowserStack Azure DevOps extension embeds your Test Reporting & Analytics build report into the Azure Pipelines run summary, so you can review pass and fail counts, failure categories, and test-level detail without leaving Azure DevOps.

In this guide, you’ll learn how to:

• Install the BrowserStack Azure DevOps extension
• Configure BrowserStack in Azure Pipelines
• Configure BrowserStack Local in Azure Pipelines
• Embed the build report in Azure Pipelines
• View the build report
• Rerun failed tests

Prerequisites

Before you start, ensure that:

• You have a BrowserStack Test Reporting & Analytics account, and your tests already report build data to it.
• You’ve hosted your project on any of the source repositories supported by Azure.
• You are a maintainer or owner of the project.
• You are an administrator of your Azure DevOps organization.
• You’ve created a pipeline for the project’s repository. If you are yet to create a pipeline, check out how to create an Azure Pipeline.
• You have access to the BrowserStack account credentials, namely access key and username.

Install the BrowserStack Azure DevOps extension

The BrowserStack Azure DevOps extension streamlines configuring Azure Pipelines to run tests on BrowserStack and to publish reports back to the pipeline. To install the extension, follow these steps:

1. In the Visual Studio Marketplace, click Azure DevOps.
2. Search for BrowserStack Azure DevOps Extension.
3. In the BrowserStack Azure DevOps Extension page, click Get it Free. The Sign in page opens.
4. Sign in to your Azure account, click Select an Azure DevOps organization and select your organization.
5. Click Install to add the extension.

BrowserStack’s extension is now installed in your Azure DevOps organization and the BrowserStack Config, BrowserStackResults, and BrowserStackStopLocal tasks are added to the list of available tasks in your pipeline.

Configure BrowserStack in Azure Pipelines

For your tests to run on BrowserStack and report to Test Reporting & Analytics, the Azure pipeline needs additional information, such as your BrowserStack credentials. You supply the credentials through a BrowserStack service connection and the BrowserStack Config task.

You can create a pipeline using either a YAML editor or a classic editor. Depending on the editor you used to create the pipeline, refer to the steps in the following tabs:

   - task: BrowserStackConfig@0
     inputs:
       browserstackServiceEndPoint: 'browserstack-connection'

Your build pipeline is successfully configured to connect with the BrowserStack cloud for your project.

Configure BrowserStack Local in Azure Pipelines

Use BrowserStack Local Testing to test your web applications hosted on private, development, and internal servers. With the BrowserStack Local Testing binary, you can create a secure, private connection between the BrowserStack Selenium Grid and your internal servers.

If your application is publicly accessible, skip this section and go to Embed the build report in Azure Pipelines.

Configuring BrowserStack Local involves:

• Set variables for BrowserStack Azure DevOps extension
• Edit your test script to call environment variables
• Add task to start BrowserStack Local
• Set proxy URL in the pipeline
• Add task to stop BrowserStack Local

Set variables for BrowserStack Azure DevOps extension

The BrowserStack Azure DevOps extension, by default, sets the following environment variables:

BROWSERSTACK_USERNAME
BROWSERSTACK_ACCESS_KEY
BROWSERSTACK_LOCAL
BROWSERSTACK_LOCAL_IDENTIFIER
BROWSERSTACK_BUILD_NAME

When you create a service connection, the BROWSERSTACK_USERNAME and the BROWSERSTACK_ACCESS_KEY are automatically added as variables during the configuration step.

In addition, when you add the BrowserStack Config task in your pipeline, the BROWSERSTACK_LOCAL and BROWSERSTACK_LOCAL_IDENTIFIER environment variables are set automatically by the extension.

The extension also adds the BROWSERSTACK_BUILD_NAME variable that autogenerates a build name for your test runs. The BrowserStack Results task uses this build name to fetch the matching Test Reporting & Analytics build.

Edit your test script to call environment variables

Edit your test script to add the environment variables for setting capabilities using the following code snippets:

String username = System.getenv("BROWSERSTACK_USERNAME");
String accessKey = System.getenv("BROWSERSTACK_ACCESS_KEY");
String buildName = System.getenv("BROWSERSTACK_BUILD_NAME");
String local = System.getenv("BROWSERSTACK_LOCAL");
String Localidentifier = System.getenv("BROWSERSTACK_LOCAL_IDENTIFIER");

MutableCapabilities capabilities = new MutableCapabilities();
capabilities.setCapability("browserName", "Chrome");
capabilities.setCapability("browserVersion", "100.0");
HashMap<String, Object> browserstackOptions = new HashMap<String, Object>();
browserstackOptions.put("os", "Windows");
browserstackOptions.put("osVersion", "10");
browserstackOptions.put("sessionName", "BStack Build Name: " + buildName); // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
browserstackOptions.put("local", local);
browserstackOptions.put("localIdentifier", localIdentifier);
browserstackOptions.put("seleniumVersion", "4.0.0");
capabilities.setCapability("bstack:options", browserstackOptions);

WebDriver driver = new RemoteWebDriver(new URL("https://" + username + ":" + accessKey + "@hub.browserstack.com/wd/hub"), capabilities);

username = process.env.BROWSERSTACK_USERNAME
accessKey = process.env.BROWSERSTACK_ACCESS_KEY
buildName = process.env.BROWSERSTACK_BUILD_NAME
local = process.env.BROWSERSTACK_LOCAL
localIdentifier = process.env.BROWSERSTACK_LOCAL_IDENTIFIER

var capabilities = {
  'bstack:options' : {
    "os" : "Windows",
    "osVersion" : "10",
    "sessionName" : "BStack Build Name: " + buildName, // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
    "local" : local,
    "localIdentifier" : localIdentifier,
    "userName" : username,
    "accessKey" : accessKey,
    "seleniumVersion" : "4.0.0",
  },
    "browserName" : "Chrome",
    "browserVersion" : "100.0",
}

var driver = new webdriver.Builder().
  usingServer("https://hub-cloud.browserstack.com/wd/hub").
  withCapabilities(capabilities).
  build();

String username = Environment.GetEnvironmentVariable("BROWSERSTACK_USERNAME");
String accessKey = Environment.GetEnvironmentVariable("BROWSERSTACK_ACCESS_KEY");
String buildName = Environment.GetEnvironmentVariable("BROWSERSTACK_BUILD_NAME");
String local = Environment.GetEnvironmentVariable("BROWSERSTACK_LOCAL");
String localIdentifier = Environment.GetEnvironmentVariable("BROWSERSTACK_LOCAL_IDENTIFIER");

ChromeOptions capabilities = new ChromeOptions();
capabilities.BrowserVersion = "100.0"; // Use latest-beta, latest, latest-1 and so on
Dictionary<string, object> browserstackOptions = new Dictionary<string, object>();
browserstackOptions.Add("os", "Windows");
browserstackOptions.Add("osVersion", "10");
browserstackOptions.Add("sessionName", "BStack Build Name: " + buildName); // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
browserstackOptions.Add("userName", username);
browserstackOptions.Add("accessKey", accessKey);
browserstackOptions.Add("seleniumVersion", "4.0.0");
browserstackOptions.Add("local", local);
browserstackOptions.Add("localIdentifier", localIdentifier);
capabilities.AddAdditionalOption("bstack:options", browserstackOptions);

driver = new RemoteWebDriver(new Uri("https://hub.browserstack.com/wd/hub/"), capabilities);

$username = getenv('BROWSERSTACK_USERNAME');
$accessKey = getenv('BROWSERSTACK_ACCESS_KEY');
$buildName = getenv('BROWSERSTACK_BUILD_NAME');
$local = getenv('BROWSERSTACK_LOCAL');
$localIdentifier = getenv('BROWSERSTACK_LOCAL_IDENTIFIER');

$caps = array(
'bstack:options' => array(
    "os" => "Windows",
    "osVersion" => "10",
    "sessionName" => "BStack Build Name: ".$buildName, // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
    "browserstack.local" => "".$local,
    "browserstack.localIdentifier" => "".$localIdentifier
    "seleniumVersion" => "4.0.0",
),
  "browserName" => "Chrome",
  "browserVersion" => "100.0",
);
$web_driver = RemoteWebDriver::create(
  "https://".$username.":".$accessKey."@hub.browserstack.com/wd/hub",
  $caps
);

username = os.environ.get('BROWSERSTACK_USERNAME')
accessKey = os.environ.get('BROWSERSTACK_ACCESS_KEY')
buildName = os.environ.get('BROWSERSTACK_BUILD_NAME')
local = os.environ.get('BROWSERSTACK_LOCAL')
localIdentifier = os.environ.get('BROWSERSTACK_LOCAL_IDENTIFIER')

bstack_options = {
    "os" : "Windows",
    "osVersion" : "10",
    "sessionName" : "BStack Build Name: " + buildName, # CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
    "local": local,
    "localIdentifier": localIdentifier,
    "seleniumVersion" : "4.0.0",
    "userName": username,
    "accessKey": accessKey
}
options = ChromeOptions()
options.set_capability('bstack:options', bstack_options)
driver = webdriver.Remote(
    command_executor="https://hub.browserstack.com/wd/hub",
    options=options)

username = ENV['BROWSERSTACK_USERNAME']
accessKey = ENV['BROWSERSTACK_ACCESS_KEY']
buildName = ENV['BROWSERSTACK_BUILD_NAME']
local = ENV['BROWSERSTACK_LOCAL']
localIdentifier = ENV['BROWSERSTACK_LOCAL_IDENTIFIER']

# to run on Chrome
options = Selenium::WebDriver::Options.chrome
capabilities = {
  'bstack:options' => {
    "os" => "Windows",
    "osVersion" => "10",
    "sessionName" => "BStack Build Name: " + buildName, # CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
    "local" = local,
    "localIdentifier" = localIdentifier,
    "seleniumVersion" => "4.0.0",
  },
  "browserName" => "Chrome",
  "browserVersion" => "100.0",
}

options.add_option('bstack:options', bstack_options)
driver = Selenium::WebDriver.for(:remote,
  :url => "https://"+username+":"+accessKey+"@hub.browserstack.com/wd/hub",
  :capabilities => options)

String username = System.getenv("BROWSERSTACK_USERNAME");
String accessKey = System.getenv("BROWSERSTACK_ACCESS_KEY");
String browserstackLocal = System.getenv("BROWSERSTACK_LOCAL");
String buildName = System.getenv("BROWSERSTACK_BUILD_NAME");
String browserstackLocalIdentifier = System.getenv("BROWSERSTACK_LOCAL_IDENTIFIER");

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("os", "Windows");
capabilities.setCapability("browser", "chrome");
capabilities.setCapability("build", buildName); // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
capabilities.setCapability("browserstack.local", browserstackLocal);
capabilities.setCapability("browserstack.localIdentifier", browserstackLocalIdentifier);
  
driver = new RemoteWebDriver(new URL("https://" + username + ":" + accessKey + "@hub.browserstack.com/wd/hub"), capabilities);

username = process.env.BROWSERSTACK_USERNAME
accessKey = process.env.BROWSERSTACK_ACCESS_KEY
var buildName = process.env.BROWSERSTACK_BUILD_NAME;
browserstackLocal = process.env.BROWSERSTACK_LOCAL
browserstackLocalIdentifier = process.env.BROWSERSTACK_LOCAL_IDENTIFIER
  
var capabilities = {
 "os" : "Windows",
 "browser" : "chrome",
 	"build" : buildName, // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
 "browserstack.local" : browserstackLocal,
 "browserstack.localIdentifier" : browserstackLocalIdentifier,
 "browserstack.user" : username,
 "browserstack.key" : accessKey
}
  
var driver = new webdriver.Builder().
  usingServer("https://hub-cloud.browserstack.com/wd/hub").
  withCapabilities(capabilities).
  build();

username = Environment.GetEnvironmentVariable("BROWSERSTACK_USERNAME");
accessKey = Environment.GetEnvironmentVariable("BROWSERSTACK_ACCESS_KEY");
buildName = Environment.GetEnvironmentVariable("BROWSERSTACK_BUILD_NAME");
browserstackLocal = Environment.GetEnvironmentVariable("BROWSERSTACK_LOCAL");
browserstackLocalIdentifier = Environment.GetEnvironmentVariable("BROWSERSTACK_LOCAL_IDENTIFIER");
  
IWebDriver driver;
OpenQA.Selenium.Chrome.ChromeOptions capability = new OpenQA.Selenium.Chrome.ChromeOptions();
capability.setCapability("os", "Windows", true);
capability.AddAdditionalCapability("os_version", "10", true);
capability.AddAdditionalCapability("browser", "Chrome", true);
capability.AddAdditionalCapability("browser_version", "latest", true); // Use latest-beta, latest, latest-1 and so on
capability.AddAdditionalCapability("name", "BStack-[Azure] Sample Test", true); // test name
capability.AddAdditionalCapability("build", buildName, true); // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
capability.AddAdditionalCapability("browserstack.local", browserstackLocal, true);
capability.AddAdditionalCapability("browserstack.localIdentifier", browserstackLocalIdentifier, true);
capability.AddAdditionalCapability("browserstack.user", username, true);
capability.AddAdditionalCapability("browserstack.key", accessKey, true);
  
driver = new RemoteWebDriver(new Uri("https://hub-cloud.browserstack.com/wd/hub/"), capability);

$username = getenv("BROWSERSTACK_USERNAME");
$accessKey = getenv("BROWSERSTACK_ACCESS_KEY");
$buildName = getenv("BROWSERSTACK_BUILD_NAME");
$browserstackLocal = getenv("BROWSERSTACK_LOCAL");
$browserstackLocalIdentifier = getenv("BROWSERSTACK_LOCAL_IDENTIFIER");
  
$caps = array(
  "os" => "Windows",
  "browser" => "chrome",
  "build" => $buildName, // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
  "browserstack.local" => $browserstackLocal,
  "browserstack.localIdentifier" => $browserstackLocalIdentifier
);
  
$web_driver = RemoteWebDriver::create(
  "https://" . $username . ":" . $accessKey . "@hub-cloud.browserstack.com/wd/hub",
  $caps
);

username = os.getenv("BROWSERSTACK_USERNAME")
access_key = os.getenv("BROWSERSTACK_ACCESS_KEY")
build_name = os.getenv("BROWSERSTACK_BUILD_NAME")
browserstack_local = os.getenv("BROWSERSTACK_LOCAL")
browserstack_local_identifier = os.getenv("BROWSERSTACK_LOCAL_IDENTIFIER")

caps = {
 'os': 'Windows',
 'browser': 'chrome',
 'build': build_name, # CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
 'browserstack.local': browserstack_local,
 'browserstack.localIdentifier': browserstack_local_identifier,
 'browserstack.user': username,
 'browserstack.key': access_key
}

driver = webdriver.Remote(
    command_executor='https://hub-cloud.browserstack.com/wd/hub',
    desired_capabilities=caps)

username = ENV["BROWSERSTACK_USERNAME"]
access_key = ENV["BROWSERSTACK_ACCESS_KEY"]
build_name = ENV["BROWSERSTACK_BUILD_NAME"]
browserstack_local = ENV["BROWSERSTACK_LOCAL"]
browserstack_local_identifier = ENV["BROWSERSTACK_LOCAL_IDENTIFIER"]
  
caps = Selenium::WebDriver::Remote::Capabilities.new
caps["os"] = "Windows"
caps["browser"] = "chrome"
caps["build"] = build_name # CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
caps["browserstack.local"] = browserstack_local
caps["browserstack.localIdentifier"] = browserstack_local_identifier
caps["browserstack.user"] = username
caps["browserstack.key"] = YOUR_ACCESS_KEY
  
driver = Selenium::WebDriver.for(:remote,
  :url => "https://hub-cloud.browserstack.com/wd/hub",
  :desired_capabilities => caps)

Add task to start BrowserStack Local

When you add the BrowserStack Config task, you can select BrowserStack Local to run your tests on any development or staging environment. Based on the editor you used to create your pipeline, refer to the steps in the following tabs:

Set proxy URL in the pipeline

If your application is behind a proxy setup, you can specify the proxy host and port details as environment variables in Azure DevOps. Adding this information as variables ensures that the Local Testing binary passes test traffic through the proxy.

Based on the type of proxy, either HTTP or HTTPS, you can set the value for HTTP_PROXY or HTTPS_PROXY, respectively, as shown:

HTTP_PROXY = "http://<proxy-host>:<proxy-port>"

HTTPS_PROXY = "https://<proxy-host>:<proxy-port>"

If the proxy requires authentication, pass the credentials in the format of a basic auth URL, as follows:

HTTP_PROXY = "http://<username>:<password>@<proxy-host>:<proxy-port>"

HTTPS_PROXY = "https://<username>:<password>@<proxy-host>:<proxy-port>"

Depending on the editor you used to create your pipeline, refer to the steps in the following tabs:

Set additional configuration options

With the Local binary, you can set additional configurations, such as verbose logging, excluding hosts, and so on. To set any additional configuration options, add the flags in the BrowserStackLocal Options section. For example, to enable verbose logging and to route all traffic via the proxy, set the following flags in the BrowserStackLocal Options field:

{"verbose": true, "forceProxy": true}

Check out Flag for Local Binary to learn about more Local Testing parameters.

Add task to stop BrowserStack Local

Stop the BrowserStack Local binary after your tests run to free up your resources. Add the BrowserStack Stop Local task in your pipeline using the following steps:

Embed the build report in Azure Pipelines

The BrowserStack Results task fetches the Test Reporting & Analytics report for the current build and publishes it to the run summary. Depending on the editor you used to create your pipeline, follow the steps in the relevant tab:

   - task: BrowserStackResults@0
     inputs:
       browserstackProduct: 'observability'

View the build report

After the pipeline run finishes, follow these steps to view the report:

1. In your Azure DevOps project, click Pipelines.
2. Select your pipeline, and then select the run you want to review.
3. In the run summary, select the BrowserStack tab.

The tab shows the Test Reporting & Analytics report for that build, including the build status, the number of tests that passed and failed, the failure categories, and the list of failed tests.

To open the full report, select the build name in the report. The Test Reporting & Analytics dashboard opens for that build, where you can review test-level logs, failure analysis, and history.

Rerun failed tests

After you integrate Azure Pipelines, you can rerun failed tests directly from the Test Reporting & Analytics dashboard instead of triggering the whole pipeline again. For the steps, see Rerun tests on Azure.