WEBVTT

1
00:00:00.000 --> 00:00:04.000
Hey there! Welcome to this walkthrough of the Quality Gate feature in Test Observability.

2
00:00:04.000 --> 00:00:08.000
Now, let's start setting up the Quality Gate by heading to Settings.

3
00:00:08.000 --> 00:00:14.000
Then click on Quality Gate. Turn the Quality Gate toggle on to ensure the analysis runs when your

4
00:00:14.000 --> 00:00:18.000
build completes. You can also override your build status with the Quality Gate status.

5
00:00:18.000 --> 00:00:22.000
Now we can configure some quality profiles. Quality profiles are groups of rules.

6
00:00:22.000 --> 00:00:26.000
A build can have multiple quality profiles, each with its own set of rules,

7
00:00:26.000 --> 00:00:31.000
rolling up into a single Quality Gate result. Let's check out the Browser Stack default

8
00:00:31.000 --> 00:00:35.000
quality profile and out-of-the-box profile which will help you get started faster.

9
00:00:35.000 --> 00:00:41.000
This is a collection of best practice rules based on inputs from millions of test data points on

10
00:00:41.000 --> 00:00:45.000
the Test Observability platform and feedback from countless users. Let's create a custom

11
00:00:45.000 --> 00:00:51.000
quality profile now by clicking on Create. Choose a name. Then we can set up the targeting for this

12
00:00:51.000 --> 00:00:56.000
profile. A quality profile can target all projects from your organization or just your current project.

13
00:00:56.000 --> 00:01:01.000
This can help you enforce org-wide standards or run checks specific to your project. The quality

14
00:01:01.000 --> 00:01:06.000
profile can also be targeted at specific builds, either by custom build tags or by build name.

15
00:01:06.000 --> 00:01:09.000
Let's have this new quality profile target my regression build.

16
00:01:09.000 --> 00:01:14.000
Now we can start adding rules. You can choose from more than 10 customizable rules,

17
00:01:14.000 --> 00:01:19.000
from ML-powered failure category thresholds to test status checks. This can help you set up

18
00:01:19.000 --> 00:01:23.000
precise and thorough quality standards for every build. For example, the Test Executions

19
00:01:23.000 --> 00:01:28.000
Threshold rule allows me to ensure a minimum number of tests have run. This type of rule

20
00:01:28.000 --> 00:01:32.000
helps me avoid false positives by ensuring all important scenarios have actually been checked.

21
00:01:32.000 --> 00:01:36.000
Let's add some more rules. Test Observability automatically detects flaky tests, new failures,

22
00:01:36.000 --> 00:01:42.000
or other problem tests in your builds. With the Smart Tags Threshold rule, I can set a threshold

23
00:01:42.000 --> 00:01:47.000
for these problem tests. You can even prevent error regressions by adding the New Unique Errors

24
00:01:47.000 --> 00:01:53.000
rule. This can check if a test failed with a new error that you haven't seen before. This is a

25
00:01:53.000 --> 00:01:58.000
feature unique to Test Observability and helps you use errors as a dimension in evaluating

26
00:01:58.000 --> 00:02:04.000
builds. Let's take a look at how you can customize each rule further. You can customize the scope

27
00:02:04.000 --> 00:02:11.000
of each rule to apply to a whole build, or just a specific test within a build. You can do this

28
00:02:11.000 --> 00:02:16.000
by clicking on any tests or overall build in a rule, and then clicking on Selected Tests.

29
00:02:16.000 --> 00:02:22.000
You can filter tests by custom test tags or even by folder or file path. Let's configure this rule

30
00:02:22.000 --> 00:02:27.000
to target a specific class. Once you're done filtering, click on Apply. Another handy feature

31
00:02:27.000 --> 00:02:32.000
is the ability to set up a Passes If or Fails If type of quality profile.

32
00:02:32.000 --> 00:02:38.000
A Passes If type profile requires every single rule to be true to pass. This type of profile can be

33
00:02:38.000 --> 00:02:44.000
used to ensure that builds meet a set of baseline requirements. On the other hand, the Fails If

34
00:02:44.000 --> 00:02:50.000
type profile fails even if a single rule is true. This type of profile can be used to look for

35
00:02:50.000 --> 00:02:55.000
breaches in your build that are unacceptable for you. Now we're done adding all our rules,

36
00:02:55.000 --> 00:03:00.000
we can click on Create. Your Quality Gate is now ready to help you start automating the

37
00:03:00.000 --> 00:03:05.000
build verification process. Now let's look at how you can integrate the result of the Quality Gate

38
00:03:05.000 --> 00:03:10.000
into your CICD workflows. You can access the Quality Gate result via an API. The API returns

39
00:03:10.000 --> 00:03:15.000
the Quality Gate status and a detailed breakdown of the quality profiles and rules for each build.

40
00:03:15.000 --> 00:03:20.000
To help you get started even faster, we've collated some example scripts in our documentation.

41
00:03:20.000 --> 00:03:24.000
These can help you interpret the Quality Gate result and automate your pipelines.

42
00:03:24.000 --> 00:03:29.000
Let's see this in action. I've got the Quality Gate result integrated with my Jenkins pipeline.

43
00:03:29.000 --> 00:03:32.000
Let's run a build and see what happens.

44
00:03:32.000 --> 00:03:38.000
After a minute or so, I can see that my JUnit test step has passed. However, the Quality Gate

45
00:03:38.000 --> 00:03:44.000
check has failed and caused the pipeline job to be marked as failed. Now let's head to test

46
00:03:44.000 --> 00:03:49.000
observability and see what happened. I can see that the build has been marked as failed because

47
00:03:49.000 --> 00:03:54.000
my Quality Gate status has overridden the build status. Let's check out what happened by clicking

48
00:03:54.000 --> 00:04:00.000
on the run. As you can see, all 86 tests in this build have passed, but the quality profile we

49
00:04:00.000 --> 00:04:05.000
created has failed. It appears that the flakiness threshold we set has been breached, causing the

50
00:04:05.000 --> 00:04:10.000
Quality Gate to fail. We could have accidentally deployed some faulty code if we hadn't checked

51
00:04:10.000 --> 00:04:16.000
this first. Now we can debug these tests in test observability and check if this build is good

52
00:04:16.000 --> 00:04:21.000
to go. With the Quality Gate feature, you can effectively minimize the time you spend verifying

53
00:04:21.000 --> 00:04:27.000
a build and push code to production faster. You can also use the Quality Gate in a number of

54
00:04:27.000 --> 00:04:32.000
creative use cases from preventing low priority test cases blocking a pipeline to rolling back

55
00:04:32.000 --> 00:04:37.000
production deployments if checks fail after release. Thank you for watching. We're excited to see what

56
00:04:37.000 --> 00:04:40.000
you'll build with Quality Gate and test observability.

