Diagnose inconsistencies in issue counts

Root cause and explanation

If you or your team has deployed any changes to your website, the issues detected on your website can change. This could happen if there are modifications to content, HTML, CSS, or scripts.

Which products show changes to issue count after website updates

All products in BrowserStack Accessibility Testing can show changes to issue count if you updated your website.

These products are:

• Workflow Analyzer
• Website Scanner
• Automated tests

Remediation

Any updates to your website can result in differences in its accessibility issue count. This is an expected outcome as changes might have fixed old issues or introduced new ones.

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count:

• Check with your team about any recent deployments, CMS content updates, or frontend changes between scans.
• Check if any additional issues reported are in the components that were recently changed.
• Fix any new issues by following the instructions provided.

Root cause and explanation

The Workflow Analyzer captures real-time DOM states, including dynamic and user-interacted content. So it can detect more accessibility issues than static scans such as website scans. A difference in the issue count is expected if you compare scans between multiple products.

Which products show changes to issue count

You might see a few changes to issue count between the Website Scanner reports and Workflow Analyzer reports.

Root cause and explanation

Differences in issue counts between two Workflow Analyzer reports of apparently the same workflow can happen due to the following reasons:

• Workflow variations: Scans using the browser extension capture real-time DOM states including dynamic and user-interacted content. Minor differences in the workflow between two scans can cause differences in issue count.

• Scans aborted due to new page state: When page states change, the scan can take some time to complete. The prompt “Processing changes, stay on page” marks this state. Interacting with the page or going to another URL when this prompt is displayed can abort ongoing scans and impact the number of issues reported.

• Scans completed abruptly: After you click “Save Report”, it can take up to 25 seconds for the extension to complete post-processing of the scan.

Which products show changes to issue count between different scans of the same workflow

Workflow Analyzer

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count:

• Workflow variations: When you retest, repeat the exact same workflow to compare with previous scans.

• Scans aborted due to new page state: Wait for the “Scanner on Standby” state to appear in the extension.

• Scans completed abruptly: Wait up to 25 seconds after you click “Save Report”.

Root cause and explanation

Changing scan settings for WCAG version, Advanced Rules, Best Practices, and Potential issue can impact issue detection. If multiple scans run with different settings, a difference in issue count is expected.

Which products show inconsistencies in issue count between different scan options

All products in BrowserStack Accessibility Testing can show inconsistencies in issue count if you run scans with different settings.

These products are:

• Workflow Analyzer
• Website Scanner
• Automated tests

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count:

• Compare scan settings and configuration between runs to ensure consistency.

• For Website Scanner, also check the local and authentication configurations.

Additional resources

• Configure scan settings on Workflow Analyzer
• Configure scan settings on Website Scanner
• Configure local testing for Website Scanner
• Configure authorization for Website Scanner
• Configure scan settings on Automated tests

Root cause and explanation

When you hide an issue, BrowserStack Accessibility Testing excludes it from the issue count. If you later unhide the issue, it is counted again. So, if you hide or unhide issues between two scans, the issue count differs.

Which products show changes to issue count due to hidden issues

All products in BrowserStack Accessibility Testing can show changes to issue count if you have hidden or unhidden issues between two scans.

These products are:

• Workflow Analyzer
• Website Scanner
• Automated tests

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count:

• Check if you have hidden issues that are legitimate accessibility issues. If so, consider unhiding them.
• Check whether you unhid an issue that was hidden earlier. Double-check whether such issues need to be hidden.

Root cause and explanation

If any page failed to load during a scan due to 404 errors, timeouts, or similar issues, it can cause differences in the issues detected. Pages that do not load are not scanned. The report then shows fewer issues.

Which products show changes to issue count due to page failure

The following products in BrowserStack Accessibility Testing can show changes to issue count if there were page failures:

• Website Scanner
• Automated tests

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count:

• Check for flaky pages on your website that frequently face timeouts and 404 errors.
• Consider improving load times of your webpages by following best practices.
• Fix 404 errors on your website by adding redirects.

Root cause and explanation

Network drops can cause website load failure and lead to differences in issues reported.

If your VPN does not allow external connections, or if it does not allow web-socket connections, differences in issue counts can occur.

Which products show changes to issue count due to network failure

All products in BrowserStack Accessibility Testing can show changes to issue count due to network failure or the impact of a VPN.

These products are:

• Workflow Analyzer
• Website Scanner
• Automated tests

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count:

• Perform accessibility scans on a stable network.
• Consider reconfiguring your VPN when running accessibility scans.

Root cause and explanation

The Spectra™ Rule Engine, which works under the hood of BrowserStack Accessibility Testing, is updated regularly to improve issue detection. There could be differences in the number of issues or type of issues detected if you compare scans run on different versions of the Spectra™ Rule Engine.

Which products show inconsistencies in issue count due to rule engine update

All products in BrowserStack Accessibility Testing can show inconsistencies in issue count if there were updates to the Spectra™ Rule Engine.

These products are:

• Workflow Analyzer
• Website Scanner
• Automated tests

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count due to an update to the Spectra™ Rule Engine:

• Check the All Issues tab to see if any of the changes in issue counts are due to the issues marked as new rules.
• Check the scanner version used for each scan.
• Review the rule update release log.

Additional resources

• Spectra Rule Engine Release Notes

Root cause and explanation

Dynamic content like ads, popups, and carousels can cause a website to behave differently at different times. Even cookie consent banners and network speed might cause changes in a website. Such differences during different scan runs can cause changes to issue count reported by BrowserStack Accessibility Testing.

Which products show changes to issue count due to dynamic content

All products in BrowserStack Accessibility Testing can show changes to issue count if dynamic content exists on your website.

These products are:

• Workflow Analyzer
• Website Scanner
• Automated tests

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count:

• If you have a cookie consent banner on your website, and if you are using Website Scanner, add Cookie Configuration settings when you create a new scan, to avoid differences due to cookie settings.
  
• Manually trigger popups, modals, and dynamic sections before rescanning if needed.
• If possible, lock dynamic content to a static state before scanning.
• If these steps do not work, consider ignoring differences due to dynamic content.

Root cause and explanation

Geolocation-based content, A/B test variations, or session-based personalization can cause different pages or elements to load during scans. This can cause an inconsistency in the number of issues reported.

Which products show changes to issue count due to personalized content

All products in BrowserStack Accessibility Testing can show changes to issue count if personalized content exists on your website.

These products are:

• Workflow Analyzer
• Website Scanner
• Automated tests

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, follow these steps to avoid changes to issue count:

• Check if personalization scripts or testing frameworks were active during the scans.
• Consider standardizing scan conditions or disabling A/B testing when you test accessibility.

Root cause and explanation

If none of the other reasons listed explains the difference, the root cause needs deeper analysis. If you think that there is a mismatch in issue count because of a reason not listed above, the support team might need to review the scans manually.

Remediation

Retry the scan to check whether the issue persists.

If the issue persists, submit both scans that show changes to issue count for manual review through the support escalation process.

Additional resources

• Contact support