Data mining API calls affect crawling budgets



The source – twitter.com

Google search employee John Mueller was asked a question about a site that communicates with an API on a client site to retrieve data – whether those URLs are factored into the crawling budget, and whether using the disallow directive could lead to problems.

John Mueller replied that the calls that are included in the page rendering process will be factored into the crawl budget. They need to be scanned to render the page.

Scanning API/URL data can be blocked using the disallow directive (if it consumes a lot of resources), this can affect content rendering. If these URLs are necessary for successful rendering, then the data should not be blocked from scanning. If they are auxiliary (drawings, diagrams, etc.), then the risks are minimal.

Share

Spelling error report

The following text will be sent to our editors: