The issue at hand pertains to URLs that possess a nofollow directive solely within the HTTP response HTML rather than being present in the rendered HTML.
Why is this important?
Webpage rendering demands substantial resources and time compared to simply retrieving the source HTML. Google, therefore, approaches URL crawling in two phases: a preliminary assessment of the response HTML and a subsequent appraisal post-rendering. The search index is then adjusted based on the post-render findings.
At the 'Processing' phase, activities unfold simultaneously, introducing complexity as Google determines which pages to render based on meta robots checks. During this phase, the response HTML is examined for followable links that are fed back into the crawl queue. This happens prior to the rendering phase.
A nofollow directive in the response HTML could decelerate the discovery and indexing of linked URLs by Google.
Moreover, inconsistency in nofollow indications raises ambiguity—should the nofollow also be reflected in the rendered HTML or not?
What does the Optimization check?
Loud Interactive's Optimization is activated by any internal URL that carries a nofollow robots tag in the response HTML but lacks it in the rendered HTML.
Examples that trigger this Optimization
Consider a circumstance where an internal URL has a meta robots=nofollow tag in the <head> of the response HTML but fails to show it in the rendered HTML.
For instance, the response HTML might appear as follows:
<!doctype html><html lang="en"><head> <meta name="robots" content="index,nofollow"> ...</head><body>...</body></html>
In contrast, the rendered HTML could be:
<!doctype html><html lang="en"><head> <meta name="robots" content="index,follow"> ...</head><body>...</body></html>
How do you resolve this issue?
Clarification regarding whether pages ought to be marked nofollow is crucial. Align the robots directives in the response HTML with those post-JavaScript rendering to ensure consistency.