How Can I Use Screaming Frog To Crawl An Entire Website?

Before starting a crawl, you should take a moment to consider what kind of information you’re looking for, how large the site is, and how much of the site you’ll need to access. Sometimes, it is a good idea to limit the crawler to a subsection of URLs when working with large areas. Consequently, data exports and file sizes are kept more manageable. This is explained in more detail below. You’ll need to make changes to the spider configuration to crawl your entire site, including all subdomains. 


What is a Website Crawl?

Typically, Screaming Frog only crawls subdomains you enter in the search bar. A spider will treat any additional subdomains as external links. The Spider Configuration menu must be changed to crawl different subdomains. The option “Crawl all subdomains” ensures that any links to other subdomains on your site encountered by the spider will be crawled.


How Can I Crawl a Subdirectory?

Enter the URL for a single folder and click start without changing any default settings. If the default settings is overwritten, reset them from the ‘File’ menu. Select the option to ‘Crawl Outside Of Start Folder’ in the Spider Configuration settings before entering your specific starting URL; if you want to start your crawl within a particular folder but crawl the rest of the subdomain.


Can I See a List of All My Website Pages?

There is a simple way to accomplish this. Screaming Frog’s default crawl setting is set to crawl all images, JavaScript, CSS, and flash files. In the Spider Configuration menu, uncheck the options’ Check Images’, ‘Check CSS’, ‘Check JavaScript’, and ‘Check SWF’ to crawl HTML only. Unchecking these settings allows the spider to contribute a list of all of the pages on your site that have internal links guided towards them. After the crawl is complete, you can filter your results by ‘HTML’ on the ‘Internal’ tab. You’ll get a CSV file with the full list after clicking ‘Export’.


Where Do I Start With A Website Crawl?

Are you looking to crawl your website? Need help from a professional? Look no further, as the experts at Boston Web Marketing can help you today. Contact us for more information by calling  (857) 526-0096 or emailing us at We look forward to hearing from you soon!

Recent Blog Posts

Contact Us Today!