Skip to main content
AI agents are now live in Website Scanner! Enter your URL and fix website issues 3x faster with AI. Explore now!
No Result Found
Connect & Get help from fellow developers on our Discord community. Ask the CommunityAsk the Community

Create a scan using Crawler

Learn how to use the crawler to extract pages from your website and create a scan.

You can crawl your website to automatically extract a list of pages, which you can then select and add to a scan. This is especially helpful if you do not have a valid sitemap or prefer a quicker way to discover pages.

Set up a new scan using Crawler

Follow these steps to create a scan using Crawler:

Access the scanner

Open the Website Scanner and click Create a new scan to get started.

Create a new scan

You can also access the Website Scanner from both Percy and Accessibility Testing.

Ensure that you have selected scanners as per your requirement.

Click the Crawler tab.

  • In the Enter page URL text field, enter the URL and click Start Crawl.

Enter domain URL using Crawler option

  • By default, the system assigns a generic title, click the Scan Name and enter a unique, descriptive identifier (see annotation 1).
  • Click the settings icon (see annotation 2) to schedule the scan and modify configurations.

Scan name & scan settings edit option

  • Enable Enable recurring scans, and enter the recurrence details.
    For more information, see how to schedule scan.

Advance scan settings combined general option

To scan a page that requires a user authentication, add Test pages behind login in Scan Settings.

  • Click the schedule text link to modify the execution frequency (see annotation 3).
  • Click the quick settings icon (see annotation 4) to modify essential crawl settings.

Quick settings combined general option

  • Update the necessary crawl settings and click Apply.

Quick settings combined general option

  • Click Start Crawl.
    Wait until the crawler lists all the urls.

  • Click Create Scan.

Create scan option

To scan a page that requires a user authentication, add Login configuration in More settings.

Scans will begin running right away if Schedule recurrent scans is not enabled.

Additional resources

We're sorry to hear that. Please share your feedback so we can do better

Contact our Support team for immediate help while we work on improving our docs.

We're continuously improving our docs. We'd love to know what you liked





Thank you for your valuable feedback

Is this page helping you?

Yes
No

We're sorry to hear that. Please share your feedback so we can do better

Contact our Support team for immediate help while we work on improving our docs.

We're continuously improving our docs. We'd love to know what you liked





Thank you for your valuable feedback!

Talk to an Expert
Download Copy Check Circle