google.com/goto: Google's anti-scraping update

If you scrape Google Search results, your existing integrations will need to be updated to parse the Location header from google.com/goto links.

Google Search has updated its organic result links, now routing them through google.com/goto?url=... instead of directly displaying the destination URL. When a user clicks a search result, Google issues a redirect to the actual webpage. This new format, which began appearing more consistently in late August 2026, particularly for logged-out or private browsing sessions, encodes the destination URL in a custom, opaque format within the "url" parameter. Unlike previous redirect wrappers like google.com/url?q=[URL], the new google.com/goto links do not expose the target URL directly in the HTML. Instead, the true destination URL is found in the "Location" header when the /goto URL is requested with a HEAD method, not by following the redirect itself. This change is part of Google's ongoing efforts to deter automated scraping of its Search Engine Results Pages (SERPs) by making it more resource-intensive to extract destination URLs.

This modification significantly complicates automated data extraction from Google Search, a common practice for AI training and competitive analysis. By obscuring the final URL and requiring an additional request to discover it, Google increases the latency and operational cost for scrapers. This move, coupled with prior restrictions like the removal of the "&num=100" parameter and enhanced bot detection, signals a determined strategy to disincentivize large-scale, unapproved harvesting of its search result data.

The necessity for scrapers to perform HEAD requests to Google's /goto URLs to retrieve the destination link from the Location header creates a more traceable and costly interaction for automated clients. This directly impacts services that build indexes or provide search data derived from SERPs. While tools like Autom have already updated their pipelines to accommodate this change by resolving the Location header, other independent scrapers and AI firms relying on direct HTML parsing will face immediate technical hurdles and increased operational expenses.