By ByteBot
In late August 2026, Google silently rewrote every organic search result link — replacing readable destination URLs with opaque google.com/goto?url=CAES... redirects that cannot be decoded locally. No announcement. No documentation. No deprecation notice. If your code extracts URLs from Google’s HTML, it broke last week. You may have just found out today.
The CAES token in the redirect parameter is a Tink-encrypted Protobuf blob. It correlates with the destination URL’s length but cannot be decrypted without Google’s private key. There is no local workaround — you have to ask Google where the link goes.
The Fix Is Simple. The Implications Are Not.
The technical workaround is short. Send a GET request (not HEAD — HEAD returns 200 without a Location header) to the /goto URL with redirect-following disabled, then read the 302 response’s Location header:
def resolve_google_goto(url: str) -> str:
response = requests.get(
url,
allow_redirects=False,
timeout=10,
)
if response.status_code != 302:
raise RuntimeError(f"Expected 302, got {response.status_code}")
return response.headers.get("Location")Deduplication matters: a single SERP page contains roughly 78 /goto anchors but only 40 unique tokens. Resolve unique tokens only. For concurrency, ScrapingBee’s benchmarks show 5 workers delivers ~40 URLs/second — anything above 10 workers increases latency due to connection contention. Tokens are session-independent: you can collect them on one IP and resolve them from another. They appear stable for at least 24 hours.
However, the performance cost compounds fast. At scale, every URL now requires an extra network round trip. Pipelines that once batch-extracted thousands of links from raw HTML in a single pass now make per-link requests back to Google — slower, more expensive, and detectable when done in sequence.
Google Is Not Targeting Every Scraper
This is not a universal rollout. Scraping.club’s analysis found Google selectively serves the goto markup to clients it fingerprints as automated. Human users on residential IPs continue to see direct destination URLs. Developers on datacenter IPs or flagged user agents get the encrypted version.
The primary targets are SERP API vendors — SerpApi, Autom, DataForSEO — whose business model is reselling Google’s index, and AI crawlers building competing search indexes. These are companies Google can frame as extracting commercial value from its index without compensation. The goto mechanism gives Google a dial: throttle their access without blocking them outright, quietly, with no public statement.
Individual developers scraping Google for research or one-off tasks are unlikely to be fingerprinted consistently. If your pipeline looks like a browser, you may still see direct URLs — for now.
The Hypocrisy Is the Point
The reaction on Hacker News hit #1 today (549 points, 440 comments) — and the top sentiment is not technical, it’s principled. Google built its empire by crawling the web with Googlebot, extracting content from third-party sites at massive scale, and indexing it without asking permission. Google’s terms of service prohibit scraping its results — terms that were never applied to Googlebot crawling everyone else’s sites.
Harvard Law School documented the contradiction directly: “Google Built Its Empire Scraping The Web. Now It’s Suing To Stop Others From Scraping Google.” The goto change is the most technically sophisticated move in that pattern yet. It is not a firewall — it is a toll booth with a one-way mirror.
Whether this constitutes anti-competitive behavior is a different question, but the developer community is asking it. Google’s Programmable Search Engine API is the approved way to access its index programmatically. The companies most disrupted by goto are direct competitors to that product.
What Developers Must Do Right Now
If you use SerpApi or Autom, you need to do nothing — both vendors deployed fixes by September 5. If you run your own pipeline, implement the resolve-via-302 pattern above with deduplication and retry logic.
For teams deciding whether to keep maintaining custom Google scrapers: factor in the ongoing cat-and-mouse costs. The Bing Search API is cheaper, has no scraping restrictions, and covers a different but substantial index. Google’s Programmable Search Engine API is rate-limited but official. Neither replaces Google’s full index, but both offer stability that rolling your own no longer does.
Key Takeaways
- Google replaced direct SERP destination URLs with Tink-encrypted google.com/goto redirects in late August 2026 — no announcement, no documentation, no timeline
- The fix: GET request with
allow_redirects=False, read the 302 Location header; use 5 workers and deduplicate tokens first - Google is selectively targeting SERP API vendors and AI crawlers — pipelines that look like browsers may still receive direct URLs for now
- SerpApi and Autom have already adapted; developers on custom pipelines must ship the resolve-via-302 fix
- This is a strategic move, not a bug fix — Google is using its index as leverage against competitors, and the mechanism can escalate
