Any seasoned SEO professional will tell you how important website crawls are for maintaining strong technical SEO.

But here lies the better questions – how frequently should you perform website crawls?

And how often are they actually performing them?

In this post, we’ll discuss what SEO publications suggest as a “best practice” web crawling cadence and the actual rate SEO pros are performing them.

Then, I’ll demonstrate the benefits of a ramped-up web crawling cadence by sharing a case study from FOX.com.

What’s a Website Crawl, Anyway?

Using specialized tools such as Screaming Frog or DeepCrawl, you can take a look “under the hood” of a website – much like a mechanic does when inspecting cars.

But instead of inspecting the mechanical parts of a car, you are inspecting the optimizable elements of a website – including the quality of its metadata, XML sitemaps, response codes, and more.

When something isn’t working as expected in SEO, it’s up to you to diagnose the problem and find the solution to fix it.

What Are the Industry Experts Saying About Website Crawl Cadence ‘Best Practice’?

Industry publications seem to be in agreement that “mini” technical audits should be conducted on a monthly basis and “in-depth” technical audits should be conducted on a quarterly or semi-quarterly basis.

However, there is little chatter specifically discussing the “optimal frequency” for performing website crawls:

• “Part of your ongoing SEO strategy includes regular audits to allow you to find and fix issues quickly (we recommend quarterly).” – Erika Varagouli, SEMrush
• “It’s good practice to do an automated scan once a month. This will be often enough to bring up major issues, like any on-page errors in the form of broken links, page titles and meta-data or duplicate content. – Digital Marketing Institute
• “I perform an SEO audit for my clients the first month, monthly (mini-audit), and quarterly (in-depth audit).” – Anna Crowe, Search Engine Journal

A website crawl and technical audit are not the same thing (thank you for the clear separation of the two, Barry Adams).

However, it’s fair to assume that these publications would recommend running a website crawl at least as frequently as they run mini-audits on a monthly basis.

What I Learned About SEO Crawl Cadence in ‘Actual Practice’

Best practice is one thing.

But how often do SEO pros run website crawls for client sites in actual practice?

To get an idea, I took to Twitter. (Yes, Twitter polls do have their obvious limitations – but it’s one of the simplest means to get some tangible data.)

Three days, and nearly 2,000 votes later, the results were in:

 

Approximately 57% of SEO pros who participated in my poll fell into the “monthly or longer” bucket, while 43% fell into the “weekly” or shorter bucket.

In other words, we were all over the map.

You may not be too surprised by these poll results.

After all, both ends of the spectrum could make complete sense – depending on the type and size of the websites you are managing.

That said, I had an experience at FOX two months ago that made me thankful that we run weekly website crawls across our major domains.

I’d like to share it with you all here – in case it encourages you to increase the cadence of your technical website crawls.

How Weekly Website Crawls Helped FOX Take Swift Action on an allRoutes.json Bug

In late July 2020, the FOX SEO team ran a routine weekly website crawl of FOX.com and discovered that 100% of our TV episode pages were unexpectedly serving error status codes (due to a bug with the allRoutes.json file).

Although the pages displayed fine for users, they were throwing 404s to Google bot – making them ineligible to appear in Google search results.

Not only is “watch time” a major KPI for the site, these pages also generate significant ad revenue for the company.

Needless to say, this crawl guided us towards a serious problem.

Thanks to the crawl we diagnosed the problem quickly.

But the solution was complex.

Over the course of three weeks (July 23 – August 13), we honed in on this specific problem and confirmed a steep decline in SEO clicks and impressions from this set of pages.

The bug was fixed by mid-August and we saw clicks and impressions trending upwards back to normal numbers.

Then, I had a thought:

If we had waited several weeks to a month before running the crawl (which is largely considered best practice), the improper response codes would have done incremental damage to the site’s SEO traffic and ad revenue.

Not only from the problem itself, but the added time to create and execute a solution.

While it’s completely possible to have diagnosed this bug without a full website crawl, it would have made the task unnecessarily difficult.

The screenshot below illustrates that without the right Google Search Console (GSC) filters being established, this particular issue hid under a veil within the Search Results report.

The GSC Coverage report was also 10 days slow to identify errors on our watch pages and only offered examples of affected URLs.

The global datasphere will grow from 33 zettabytes (each equal to a trillion gigabytes) in 2018 to 175 ZB by 2025.

In marketing, our role as the stewards of much of this data is growing, as well.

As of last year, more data is being stored in the enterprise core than in all the world’s existing endpoints, according to a report by IDC.

The great challenge for marketers and SEO professionals is activating and using that data.

In 2025, each connected person will have at least one data interaction every 18 seconds and nearly 30% of the world’s data will need real-time processing.

There’s no way human marketers can handle this processing on our own.

And more and more, as our machine-learning-enabled tools process and analyze search data, they’re learning and improving their understanding of it as they go.

Machine Learning in Search

Perhaps the best-known use of machine learning in search is Google’s own RankBrain, an algorithm that helps the search engine better understand the relevance and context of – and the relationship between – words.

Machine learning enables Google to understand the idea behind the query.

Machine learning allows the algorithm to continuously expand that understanding as new words and queries are introduced.

And as algorithms get better at determining which content best meets the needs of each searcher, we are being challenged to create content that meets those needs – and to optimize it so that relevance is clear.

It’s no coincidence that as we’re experiencing this explosion in data, interest in SEO is growing, as well.

SEO & Data Science

SEO has grown to be a viable, respectable mainstream marketing career.

As I write this, there are 823,000 people on LinkedIn with “SEO” in their profile and 8,600 who specifically categorize their core service offerings as SEO.

Looking worldwide, those figures balloon to 3.2 million and 25,000, respectively.

But this is just a small sampling of the SEO industry.

There are those in SEO who identify as content marketers, digital marketing strategists or practitioners, site developers, analytics pros, consultants, advisors, and more.

Our industry is massive in size and scope, as SEO now touches nearly every aspect of the business.

So much more is being asked of SEO professionals now, thanks to that massive increase in data we have to deal with.

Yet according to our research at BrightEdge, only 31.5% of organizations have a data scientist at their company.

Working alongside machine learning rather offers tech-savvy SEO professionals a number of important advantages.

1. Enhanced Performance in Your Field of Specialization

Employers and clients alike are driven by results.

Do you know how to use the machine-learning-powered tools in your area of specialization?

Whether in paid search, technical SEO, content creation and optimization, link building or some other facet of SEO, those who can prove superior performance through the use of machine-learning-enabled SEO tools are increasing their own value.

2. Start Ahead & Stay Ahead

Search is a live auction. If you’re waiting to see what customers think and only then getting ready to respond, you’re already behind.

Machine-learning-powered tools enable marketers to activate real-time insights, to personalize and optimize content in the moment for each users’ individual needs.

3. Economies of Scale

You are exponentially more valuable as an SEO practitioner and leader if you can demonstrate the ability to scale your efforts.

The real power of machine learning is in its ability to convert more data than we know what to do with into actionable insights and automated actions that marketers can use to really move the needle.

To do that is hard.

For example, to build BrightEdge Autopilot we had to process over 345 petabytes of data over the course of many years to help fine-tune machine learning and automated products.

Machines aren’t angling for a promotion; they don’t harbor preconceptions or care about past mistakes.

They are entirely subjective, taking opinions and personalities and other potential bottlenecks out of the process of data evaluation.

What marketers are left with are pure, accurate data outputs that can then be activated at scale to improve search visibility and interactions with customers.

4. Room to Grow

Mastering your SEO toolset gives you more room to grow in your profession, and as a person who just so happens to love the work you do.

Machine learning, in particular, empowers us to reap insights from larger datasets and gives us access to far more intelligence than when we could only learn from that we manually analyzed ourselves.

It is your specialized insight and industry knowledge that determines which outputs are useful and how they should be applied.

Machine learning can tell you very quickly how your audience’s behaviors have changed during a major market disruption, such as our recent experience with COVID-19.

But how you interpret and respond to those changes is still very much the domain of marketing and SEO professionals.

Machine learning can help you recognize patterns in visitor behavior that point to opportunities and areas in need of improvement.

What technology cannot do is replace the creative and analytical human thought process and experience that determines the best next steps to take in response to those insights.

The people of SEO cannot be replaced. In fact, they’re more important than ever.

The tools we use may be quite sophisticated; machine-learning-enabled tools can even make decisions and implement optimizations.

However, it is the people of SEO who drive the creative and analytical processes that machines simply cannot replace:

• Creative analysts.
• Data scientist (who control input into machines).
• Analytics.
• Content producers.
• Culture builders and success evangelists.
• Expert users who facilitate sales and help customers.
• Strategic planning across digital channels.

And there are agile marketers who may do any combination of the above.

They are key in facilitating collaboration with other digital departments to ensure a truly holistic SEO strategy.

In their HBR article Collaborative Intelligence: Humans and AI Are Joining Forces, H. James Wilson and Paul R. Daugherty explain the three key roles humans undertake in every interaction with machine-learning-powered technology:

• Train: We need to teach the machine to perform certain tasks.
• Explain: We must make sense of the outcome of the task, especially if it is unexpected or counterintuitive.
• Sustain: It is up to us to ensure that the technology is used logically and responsibly.

Applying this lens to our SEO tech, we see these three tenets hold true.

We need to decide which SEO tasks to intelligently automate and give our tools the proper input.

We need to take the output and make sense of it, focusing only on those insights with business-building potential.

We are responsible for ensuring that searcher privacy is protected, that the value of the technology outweighs the cost, and that it is otherwise being made good use of.

You can build your value as an SEO and learn to work more effectively with machine-learning-powered tech by building these skills:

• Data proficiency: According to Stanford researchers, the share of AI jobs grew from 0.3% in 2012 to 0.8% of total jobs posted in the U.S. in 2019. AI labor demand is growing, especially in high-tech services and the manufacturing sector.
• Communication: As the arbiter of so much customer data, it is critical that we communicate key insights and value in ways other department heads and decision-makers can understand.
• Agility: More than a trait or quality, agility is a skill developed through constant experimentation.

Embracing machine learning and automation means building synergy with human creativity and skills.

It can make us more creative and effective by uncovering SEO insights and patterns we would never have recognized otherwise.

It can help us discover new topics, identify content gaps, optimize for specific types of queries and results, and more.

What’s more, it can save vital time on tasks that are too time-consuming, too repetitive and laborious, so we can scale performance.

And as that happens, we develop new skills and progress also as part of a symbiotic relationship between people and technology.