Integrate BrowserStack Automate with Azure Pipelines

Integrate your Selenium Webdriver tests with Azure Pipelines or Visual Studio Team Services (VSTS).

Introduction

Azure Pipelines formerly known as Visual Studio Team Services(VSTS) is a Microsoft product. It is a Continuous Integration (CI) server, which automates the build and deploy process of your web applications. By running your Selenium test suite in Azure Pipelines, you also automate testing as part of the build process. Add BrowserStack to the mix, and your web apps can be tested across 2000+ real devices and desktop browsers. BrowserStack lets you integrate your Selenium tests with Azure Pipelines using an extension that connects the CI server to the BrowserStack Selenium Grid.

Important: The BrowserStack plugin is currently available only for the cloud version of Azure Pipelines.
If you are using an on-premise version of Azure DevOps kindly reach out to support.

Use this extension to:

  • Configure your BrowserStack credentials.
  • Set up and tear down the BrowserStack Local binary for testing internal, development and staging environments.
  • Embed BrowserStack test results in your job results.

Prerequisites

You need the following set up before you begin.

  • Azure Pipelines account
  • BrowserStack account

Installing Azure Pipelines extension

To run your tests with Azure Pipelines on BrowserStack, you will first need to install our extension. Before installing the plugin, ensure you have the necessary privileges to administer the installation.

  1. Go to the BrowserStack extension page.
  2. Click on Get it free. It will redirect to the installation page. browserstack automate azure plugin

  3. Use the drop-down Select an Azure DevOps organization to select your organization. Click Install. browserstack automate azure plugin

This completes the extension installation.

Configuring build pipeline

Once you have installed the extension, you can configure a build pipeline by editing an existing pipeline or creating a new one. There is a slightly different process of integrating with BrowserStack depending on the type of pipeline editing option you are using in Azure as given below:

  1. Using the YAML based pipeline editor
  2. Using the classic pipeline editor in Azure

If you already have a YAML based pipeline or want to create a new one, then you can integrate the BrowserStack extension using the steps as shown below:

  1. Open/create a project and add a new service connection for connecting with the BrowserStack cloud. Go to Project settings -> Service connections -> New service connection as shown below:

    browserstack automate azure plugin
  2. Select BrowserStack from the list, add your credentials (you can find them in account settings), give a name to the connection and save it, as shown in the image below:

    browserstack automate azure plugin
  3. Next step is to edit the pipeline in YAML mode. Create a new pipeline if you do not have an existing one. Use the BrowserStack service connection to connect to the BrowserStack cloud using the BrowserStack Config task from the right task pane in the YAML edit window as shown in the image below:

    browserstack automate azure plugin
  4. You can choose to use BrowserStack Local which allows you to test web applications hosted on private, development, and internal servers, while adding the BrowserStack Config task. Writing test script with local testing capability usage is explained in this section.
  5. Similarly, if you have enabled BrowserStack Local, you have to add the task BrowserStack Stop Local from the right-side task pane, in your YAML workflow once the test script execution is done (See sample YAML screenshot below).
  6. You can also add the BrowserStack Results task in your YAML workflow from the right-side task pane. Finally, your workflow will look somewhat similar to the workflow as shown below:

    browserstack automate azure plugin

Follow the steps outlined below to integrate with BrowserStack using the classic pipeline editor in Azure:

  1. Select your organization, project and click on Pipelines.
  2. If you want to create a new pipeline using the classic pipeline editor option, you can do so by selecting the “Use the classic editor” option as shown below:

    browserstack automate azure plugin
  3. Select Agent job 1(it is a job name, it could be different if you are editing an existing job) and click on the + icon. Type browserstack in the search box. browserstack automate azure plugin
  4. Hover on BrowserStack Config and click on the Add button.
  5. Select BrowserStack configuration setup click on the NEW button to set up BrowserStack credentials. browserstack automate azure plugin
  6. In the popup, enter a name in Connection name field. Enter your BrowserStack Username and AccessKey. You can get your credentials from the automate dashboard page. Use the Verify connection button to verify the connection. browserstack automate azure plugin
  7. Click OK to save the connection.

Configuring BrowserStack Local

BrowserStack Local Testing allows you to test web applications hosted on private, development, and internal servers. With the Local Testing binary, you can create a secure, private connection between the BrowserStack Selenium Grid and your internal servers.

The BrowserStack Azure Pipelines extension sets the following environment variables:

BROWSERSTACK_USERNAME
BROWSERSTACK_ACCESS_KEY
BROWSERSTACK_LOCAL
BROWSERSTACK_LOCAL_IDENTIFIER

Use these environment variables to set the capabilities in your tests. For example:

String username = System.getenv("BROWSERSTACK_USERNAME");
String accessKey = System.getenv("BROWSERSTACK_ACCESS_KEY");
String browserstackLocal = System.getenv("BROWSERSTACK_LOCAL");
String browserstackLocalIdentifier = System.getenv("BROWSERSTACK_LOCAL_IDENTIFIER");
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("os", "Windows");
capabilities.setCapability("browser", "chrome");
capabilities.setCapability("browserstack.local", browserstackLocal);
capabilities.setCapability("browserstack.localIdentifier", browserstackLocalIdentifier);
driver = new RemoteWebDriver(new URL("https://" + username + ":" + accessKey + "@hub.browserstack.com/wd/hub"), capabilities);
username = process.env.BROWSERSTACK_USERNAME
accessKey = process.env.BROWSERSTACK_ACCESS_KEY
browserstackLocal = process.env.BROWSERSTACK_LOCAL
browserstackLocalIdentifier = process.env.BROWSERSTACK_LOCAL_IDENTIFIER

var capabilities = {
 "os" : "Windows",
 "browser" : "chrome",
 "browserstack.local" : browserstackLocal,
 "browserstack.localIdentifier" : browserstackLocalIdentifier,
 "browserstack.user" : username,
 "browserstack.key" : accessKey
}

var driver = new webdriver.Builder().
  usingServer("https://hub-cloud.browserstack.com/wd/hub").
  withCapabilities(capabilities).
  build();
username = Environment.GetEnvironmentVariable("BROWSERSTACK_USERNAME");
accessKey = Environment.GetEnvironmentVariable("BROWSERSTACK_ACCESS_KEY");
buildName = Environment.GetEnvironmentVariable("BROWSERSTACK_BUILD_NAME");
browserstackLocal = Environment.GetEnvironmentVariable("BROWSERSTACK_LOCAL");
browserstackLocalIdentifier = Environment.GetEnvironmentVariable("BROWSERSTACK_LOCAL_IDENTIFIER");

IWebDriver driver;
OpenQA.Selenium.Chrome.ChromeOptions capability = new OpenQA.Selenium.Chrome.ChromeOptions();
capability.setCapability("os", "Windows", true);
capability.AddAdditionalCapability("os_version", "10", true);
capability.AddAdditionalCapability("browser", "Chrome", true);
capability.AddAdditionalCapability("browser_version", "latest", true); // Use latest-beta, latest, latest-1 and so on
capability.AddAdditionalCapability("name", "BStack-[Azure] Sample Test", true); // test name
capability.AddAdditionalCapability("build", buildName, true); // CI/CD job name using BROWSERSTACK_BUILD_NAME env variable
capability.AddAdditionalCapability("browserstack.local", browserstackLocal, true);
capability.AddAdditionalCapability("browserstack.localIdentifier", browserstackLocalIdentifier, true);
capability.AddAdditionalCapability("browserstack.user", username, true);
capability.AddAdditionalCapability("browserstack.key", accessKey, true);
driver = new RemoteWebDriver(new Uri("https://hub-cloud.browserstack.com/wd/hub/"), capability);
$username = getenv("BROWSERSTACK_USERNAME");
$accessKey = getenv("BROWSERSTACK_ACCESS_KEY");
$browserstackLocal = getenv("BROWSERSTACK_LOCAL");
$browserstackLocalIdentifier = getenv("BROWSERSTACK_LOCAL_IDENTIFIER");

$caps = array(
  "os" => "Windows",
  "browser" => "chrome",
  "browserstack.local" => $browserstackLocal,
  "browserstack.localIdentifier" => $browserstackLocalIdentifier
);

$web_driver = RemoteWebDriver::create(
  "https://" . $username . ":" . $accessKey . "@hub-cloud.browserstack.com/wd/hub",
  $caps
);
username = os.getenv("BROWSERSTACK_USERNAME")
access_key = os.getenv("BROWSERSTACK_ACCESS_KEY")
browserstack_local = os.getenv("BROWSERSTACK_LOCAL")
browserstack_local_identifier = os.getenv("BROWSERSTACK_LOCAL_IDENTIFIER")

caps = {
 'os': 'Windows',
 'browser': 'chrome',
 'browserstack.local': browserstack_local,
 'browserstack.localIdentifier': browserstack_local_identifier,
 'browserstack.user': username,
 'browserstack.key': access_key
}

driver = webdriver.Remote(
    command_executor='https://hub-cloud.browserstack.com/wd/hub',
    desired_capabilities=caps)
username = ENV["BROWSERSTACK_USERNAME"]
access_key = ENV["BROWSERSTACK_ACCESS_KEY"]
browserstack_local = ENV["BROWSERSTACK_LOCAL"]
browserstack_local_identifier = ENV["BROWSERSTACK_LOCAL_IDENTIFIER"]
caps = Selenium::WebDriver::Remote::Capabilities.new
caps["os"] = "Windows"
caps["browser"] = "chrome"
caps["browserstack.local"] = browserstack_local
caps["browserstack.localIdentifier"] = browserstack_local_identifier
caps["browserstack.user"] = username
caps["browserstack.key"] = access_key

driver = Selenium::WebDriver.for(:remote,
  :url => "https://hub-cloud.browserstack.com/wd/hub",
  :desired_capabilities => caps)
my $username = $ENV{"BROWSERSTACK_USERNAME"};
my $accessKey = $ENV{"BROWSERSTACK_ACCESS_KEY"};
my $browserstackLocal = $ENV{"BROWSERSTACK_LOCAL"};
my $browserstackLocalIdentifier = $ENV{"BROWSERSTACK_LOCAL_IDENTIFIER"};

my $caps = {
  "os" => "Windows",
  "browser" => "chrome",
  "browserstack.local" => $browserstackLocal,
  "browserstack.localIdentifier" => $browserstackLocalIdentifier
};

my $host = "$username:$accessKey\@hub-cloud.browserstack.com";

my $driver = new Selenium::Remote::Driver('remote_server_addr' => $host,
  'port' => '80', 'extra_capabilities' => $caps);
Note: You must pass browserstack.local and browserstack.localIdentifier capabilities to test on your local development servers.

Starting BrowserStack Local

To enable BrowserStack Local follow these steps:

  1. Under BrowserStack configuration task, click on the checkmark next to BrowserStackLocal.
  2. Use BrowserStackLocal Options to set any additional configuration options when running the binary.

For example to enable verbose logging and to force local, pass

{"verbose": true, "forceLocal": true}

in the BrowserStack Local Options. The full list of options is available here.

Please use camel case in local modifier options, for example, use logFile instead of --log-file.

Stopping BrowserStack Local

Please note that stopping BrowserStack Local is only required if you enabled Local in the above step.

If you are using the YAML pipeline editor, follow the steps below to stop BrowserStack Local:

  1. In the pipeline edit window, take the cursor to a point in the workflow where test script has completed its run.
  2. Go to the right-side tasks pane and search for BrowserStack
  3. Select BrowserStack Stop Local from the list.
  4. You will see that the following YAML will be added to the workflow

    ```yaml Settings:
    • task: BrowserStackStopLocal@0 ```
Note:
  • You should invoke BrowserStack Stop Local only if you have enabled BrowserStack Local during BrowserStack Config task setup.
  • Ensure to stop BrowserStack Local only after your test scripts have completed running

To stop BrowserStack Local follow these steps:

  1. Select Agent job 1 and click on the + icon. Type browserstack in the search box.
  2. Hover on BrowserStack Stop Local and click on the Add button.

    browserstack automate azure plugin
Note:
  • This task will stop BrowserStack Local.
  • Stop Local should only be called after the script which runs the test cases.

Generating and Embedding test reports

The BrowserStack Azure Pipelines plugin enables you to embed the test results from BrowserStack into Azure Pipelines. We currently support viewing test reports in Azure Pipelines with a select set of languages and frameworks, which are Java with JUnit or TestNG, and Node with Protractor or WebdriverIO. The sections below describe how to embed your BrowserStack test results in Azure Pipelines.

Generating test reports

Follow the below steps to generate test reports.

TestNG and JUnit

A prerequisite for embedding test reports is that the Java project must be built with Maven and use either TestNG or JUnit for testing. To configure your project:

Add the following dependencies to your pom.xml file:

<dependency>
    <groupId>com.browserstack</groupId>
    <artifactId>automate-testassist</artifactId>
    <version>1.1.1</version>
</dependency>

Add the following repositories to your pom.xml file:

<repositories>
  <repository>
    <id>sonatype-nexus-snapshots</id>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
  </repository>
</repositories>
<pluginRepositories>
  <pluginRepository>
   <id>sonatype-nexus-snapshots</id>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
  </pluginRepository>
</pluginRepositories>

Add the following plugin to your pom.xml file:

<build>
    <plugins>
      <plugin>
        <groupId>com.browserstack</groupId>
        <artifactId>automate-maven-plugin</artifactId>
        <version>1.1.1</version>
        <configuration>
            <source>${jdk.source.version}</source>
            <target>${jdk.target.version}</target>
            <complianceLevel>${jdk.source.version}</complianceLevel>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>test-compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
</build>

You must define the Maven properties, jdk.source.version and jdk.target.version to match the Java versions of your project.

To configure WebdriverIO project, add wdio-browserstack-reporter as a dependency in your package.json file.

{
  "dependencies": {
    "wdio-browserstack-reporter": "~0.1.1"
  }
}

Add browserstack as a reporter in your conf file.

// wdio.conf.js
module.exports = {
    // ...
    reporters: ['browserstack'],
    reporterOptions: {
        browserstack: {
            outputDir: './'
        }
    },
    // ...
};

outputDir (optional): defines a directory location for storing browserstack report files. Here is the link for the GitHub repository.

To configure Protractor project, add protractor-browserstack-reporter as a dependency in your package.json file.

{
  "dependencies": {
    "protractor-browserstack-reporter": "~0.1.1"
  }
}

Add browserstack as a plugin in your conf file.

// conf.js
'plugins': [{
  'package': "browserstack-protractor-reporter"
}]

outputDir (optional): defines a directory location for storing browserstack report files.

Here is the link for the GitHub repository.

Embedding test reports

If you are using the YAML pipeline editor, follow the steps below to embed reports:

  1. In the pipeline edit window, take the cursor to a point in the workflow where test script has completed its run.
  2. Go to the right-side tasks pane and search for BrowserStack
  3. Select BrowserStack Results from the list.
  4. You will see that the following YAML will be added to the workflow

    ```yaml Settings:
    • task: BrowserStackResults@0 ```

You can configure your Azure Pipelines setup to embed all the BrowserStack Selenium reports and logs:

  1. Select Agent job 1 and click on the + icon. Type browserstack in the search box.
  2. Hover on BrowserStack Results and click on the Add button. Click on Control Options and enable Continue on error.

    browserstack automate azure plugin
Note: Please make sure to add BrowserStack Results step after the step which runs your test cases.

With this integration, you can now view the results of your Selenium Webdriver tests within Azure Pipelines. This is how the test results look in Azure Pipelines: browserstack automate azure plugin

Conclusion

By following the steps outlined in this guide, you should have a seamless integration between your existing automation in Azure Pipelines, your Selenium tests and the BrowserStack Selenium grid. This will help you leverage all the benefits of test automation and the scale and coverage offered by BrowserStack.

We're sorry to hear that. Please share your feedback so we can do better






Contact our Support team for immediate help while we work on improving our docs.

We're continuously improving our docs. We'd love to know what you liked





Thank you for your valuable feedback

Is this page helping you?

Yes
No

We're sorry to hear that. Please share your feedback so we can do better






Contact our Support team for immediate help while we work on improving our docs.

We're continuously improving our docs. We'd love to know what you liked





Thank you for your valuable feedback!

Talk to automation expert