To find out what actually makes a US retailer hard to scrape, you have to separate the things that get you blocked: your IP, your TLS fingerprint, and whether you run a real browser. So we tested the 14 biggest US retail sites four different ways and watched which gate each site really enforces. The results are not what the "big name means hard" assumption would tell you.

How we tested it

We ran each of the 14 sites through four setups, changing one variable at a time.

One, a plain scripted request from a clean residential connection. Two, the same scripted request but with a real Chrome TLS fingerprint, using curl_cffi, which impersonates a browser's handshake, still a script and not a browser. Three, a real browser running on a datacenter IP. Four, the same real browser on a residential IP.

For each site we read the response headers and cookies to identify the anti-bot vendor, Akamai Bot Manager, DataDome, PerimeterX, Cloudflare, Imperva, by the fingerprints they leave behind (cookies like _abck and bm_sz for Akamai, datadome for DataDome, _px for PerimeterX). And we checked the actual page content, not just the status code, so a long "please verify you are human" page does not get counted as a success. Comparing those four results isolates exactly which gate stops you at each site.

What actually blocks you: three separate gates

Across all 14 sites, the blocking came down to three independent gates. A site can enforce one, two, or all three, and you have to pass every one it uses.

Gate one is IP reputation. The exact same real browser loaded Etsy, Wayfair, Chewy, Macy's and Kroger with no problem from a residential IP, and got blocked outright from a datacenter IP. Same browser, same everything, only the IP changed. Datacenter IPs are effectively dead for retail scraping, which is the entire reason residential proxies exist.

Gate two is your TLS fingerprint. From the same residential IP, swapping a plain request for one with a real Chrome TLS handshake unlocked Home Depot, Nike and Macy's, no browser, no new proxies, just a handshake that looked like Chrome. The fingerprint was the only thing that had been giving the scraper away. This is the cheapest, highest-leverage fix most people miss.

Gate three is browser execution. Etsy (DataDome) and Wayfair (PerimeterX) blocked even a script with a perfect Chrome TLS fingerprint on a clean residential IP. They run JavaScript challenges and behavioral checks that a script cannot answer. Only a real browser got through. So for those sites, a good IP and good TLS are necessary but not enough.

The full results

Both scripted tests ran from a residential IP. "Loads" means the real page came back; "Blocked" means a 403, 429, or challenge page; "Geo-gate" means a country-selection interstitial instead of the store.

The three tiers of US retail

Group the 14 sites by the hardest gate each one enforces, and a clear ranking appears.

Easy, where a residential IP plus a browser-grade TLS fingerprint was enough and no headless browser was needed: Costco, Walmart, Target, Lowe's, Home Depot, Nike and Macy's all returned full, real pages to a simple script once it had a Chrome fingerprint. Costco was the softest of all, running essentially no bot management. The surprise is Home Depot, which stacks three vendors (Akamai, PerimeterX and Forter) and still folded to good TLS from a clean IP.

Hard, where you need a real browser and a perfect-TLS script is not enough: eBay, Chewy, Kroger, Etsy and Wayfair. All five blocked the script but loaded in a real residential browser.

Special cases: Amazon throws its own "continue shopping" interstitial at almost everything, including a real browser, so it is aggressive but inconsistent rather than a hard wall. Best Buy showed a country-selection interstitial to our script and datacenter tests, but loaded the full US store in a real browser on a residential connection, so its wall leans on the same signals as the rest rather than a hard geo-block.

One pattern across all of it: Akamai Bot Manager is the most common vendor, on 9 of the 14 sites, but on most of them it blocks only the obvious signals, a bad TLS fingerprint or a datacenter IP. The vendors that demanded a full browser even with good TLS were DataDome (Etsy) and PerimeterX (Wayfair), plus a few stricter Akamai setups (eBay, Chewy, Kroger).

What this means if you are scraping retail

The working setup for retail at scale is all three gates at once: a residential IP, a browser-grade TLS fingerprint, and a real headless browser for the strict sites. Drop any one and a chunk of retail goes dark. This is exactly why a cheap datacenter-proxy scraper comes back with blanks on half the sites, and why "it worked on my laptop" quietly breaks in production. Your laptop had a residential IP and a real browser. Your server has neither.

It also means the difficulty is not really about the retailer. It is about how many of the three gates you are equipped to pass. A cheap self-serve tool passes zero of them well. A serious setup passes all three, and keeps passing them as the sites escalate.

That last part, keeping it working as sites change, is the real job. If you would rather have clean retail price or catalog data without building and babysitting that stack, that is what we do at DataScrape Solutions for clients. We handle the IP, the fingerprint, and the browser layer, maintain it as the retailers tighten their defenses, and hand over clean, validated data. You can start with a free sample of your exact targets so you see the data first.

FAQ

What is the hardest US retail site to scrape? In our test, the sites that blocked even a real browser on a datacenter IP and needed a full residential browser were Etsy (DataDome), Wayfair (PerimeterX), eBay, Chewy and Kroger. Amazon is also aggressive in a different way, showing its own interstitial to almost every automated request. The easiest was Costco, which runs almost no bot management.

Why does my scraper get blocked even with a good proxy? A clean IP is only one of three gates. Most scrapers get caught on their TLS fingerprint, the handshake a default HTTP client sends looks nothing like a browser, so anti-bot systems flag it before the page even loads. In our test, fixing only the TLS fingerprint (with a tool like curl_cffi) unlocked several sites from the same IP. And the strictest sites, DataDome and PerimeterX, need a real browser on top of that.

Do I need residential proxies to scrape retail sites? For the strict ones, yes. The identical browser that loaded Etsy, Wayfair, Chewy, Macy's and Kroger from a residential IP was blocked outright from a datacenter IP. Datacenter proxies are effectively dead for most retail. Easier sites like Costco or Walmart tolerate more, but a residential IP plus a browser-grade TLS fingerprint is the baseline that actually works.

Can you just deliver the retail data instead of me building this? Yes. We run the full stack, residential IPs, browser-grade fingerprints, and a real browser layer for the strict sites, maintain it as the retailers change their defenses, and hand you clean, validated data on your schedule. You can start with a free sample of your exact targets to see the quality before committing.

I kept running into this on client work, so I built a hosted API that just returns clean JSON. Here is what the hard parts actually were.

The scraping isn't the problem. Keeping it alive is.

Myntra scrapers rarely crash in a way you would notice. They drift. A layout change moves the discounted price into a different field, and suddenly every product reads as full price, no drop ever fires, and the job still finishes green. The data is wrong and nothing tells you.

On top of that, datacenter IPs get throttled or served blank responses, bursts from one address get rate-limited, and accessing the site from outside India can change what the page shows. None of it is unbeatable, but it breaks the moment Myntra ships a change, which is often. That maintenance, not the first version, is the real cost.

What the API returns

Two endpoints. Fetch a product by ID:

curl "https://myntra-product-price-data-api.p.rapidapi.com/products/35776515" \
  -H "X-RapidAPI-Key: YOUR_KEY" \
  -H "X-RapidAPI-Host: myntra-product-price-data-api.p.rapidapi.com"

It returns something like:

{
  "brand_name": "Marks & Spencer",
  "mrp": 1699,
  "discounted_price": 679,
  "coupon_discounted_price": 509,
  "available_count": 70,
  "average_rating": "4.29"
}

Prices come back as numbers, not rupee strings you have to clean up first.

The Myntra pagination gotcha (this one cost me a day)

Myntra search is cursor-paginated, and it does not tell you that anywhere. If you just increment an offset, you get page 1 back over and over, with a 200 OK, so it looks like it is working while it quietly hands you duplicates.

The trick is that each response carries a pagination cursor, and you have to walk the pages in order, feeding each response's cursor into the next request. You cannot jump straight to page 5. So the API exposes it as a proper cursor:

import requests

headers = {
    "X-RapidAPI-Key": "YOUR_KEY",
    "X-RapidAPI-Host": "myntra-product-price-data-api.p.rapidapi.com",
}
url = "https://myntra-product-price-data-api.p.rapidapi.com/search"

cursor, all_products = None, []
while True:
    params = {"keyword": "tshirts"}
    if cursor:
        params["cursor"] = cursor
    data = requests.get(url, headers=headers, params=params).json()
    all_products += data["products"]
    if not data["has_next_page"]:
        break
    cursor = data["next_cursor"]

print(len(all_products), "products")

Each call is exactly one upstream request, so no duplicate pages and no wasted calls.

Try it

It is live on RapidAPI with a free tier and no setup: https://rapidapi.com/king052/api/myntra-product-price-data-api

If you have fought Myntra, Ajio, or Nykaa scraping before, I would like to hear how you handled the Akamai side.

I took the top 1,000 sites from the Tranco ranking, fetched each one's robots.txt on October 2, 2026, and read what it said about the main AI crawlers. Here is what came back.

What "the top 1,000" actually is

One caveat up front, because it turned out to be interesting on its own. Of the 1,000 domains, only 485 served a robots.txt I could read. The rest are not websites in the normal sense. Around 320 were CDN and DNS infrastructure like akamaiedge.net and cloudfront.net. Another 68 were API endpoints such as googleapis.com with nothing to crawl. 63 returned a login page or a soft error instead of a real file, and 64 blocked the request outright. So every number below is out of the 485 sites that have a real, readable robots.txt.

About a quarter block at least one AI crawler

114 of the 485 sites, or 23.5%, block at least one of the eight major AI crawlers by name. Only 14 block all eight. So blocking happens, but blanket blocking is still rare. Most sites that block pick their targets.

Here is how often each crawler gets a full block:

Common Crawl sits at the top, and that makes sense. Its public archive feeds a lot of model training, so blocking CCBot is a way to block many AI companies at once. Bytespider, ByteDance's crawler, comes next and has a rough reputation for ignoring limits, which is probably why so many admins name it directly.

Sites also tend to treat AI crawlers as one group. Of the sites that block GPTBot, 79% also block ClaudeBot. When someone decides to close the door, they usually close it on everyone at once.

News sites are the real outlier

This is the sharp line in the data. Among the news and media sites in the set, 80% block at least one AI crawler. For everything else, it is 20%. News is four times more likely to say no.

And they are not shy about it. The New York Times, BBC, Bloomberg, CNBC, USA Today, NBC News and the Daily Mail block all eight crawlers I checked. CNN, Forbes, NPR and the Financial Times block seven.

The exceptions are the fun part. The Wall Street Journal blocks none of them by name. Neither does Fox News, Time, the Telegraph or the Independent. Reuters blocks one. Same industry, opposite bet. A lot of that comes down to licensing. A publisher that has signed a content deal with an AI company has a reason to leave the door open.

Marketplaces and platforms mostly stay open

Step outside news and the blocking drops off fast. Amazon blocks seven of the eight crawlers, but Etsy, Shopify and Booking.com block none of them by name. Wikipedia, Pinterest, IMDb, Substack and WordPress.com are wide open. Apple and Microsoft's main domains do not block them either.

So this is less about the whole web closing to AI and more about one corner of it. Newsrooms are pulling up the drawbridge. Most of the rest has not bothered yet.

64 sites would not even hand over the file

One more thing, and it matters if you pull data for a living. 64 of the 1,000 returned a 403 error to a plain request for robots.txt, a public file that exists specifically to be read. nih.gov, ScienceDirect and Stack Overflow were among them. That is a different kind of blocking, aimed at any automated request rather than any specific bot, and it is getting more common.

Why this is worth watching

If your business depends on being found or on pulling public data, this is a trend to keep an eye on. Blocking a training crawler like GPTBot is not the same as blocking the bot that fetches a page to answer a live question, and most sites that block are aiming at training, not answers. But the line keeps moving. The data a competitor publishes openly today may be closed to a crawler tomorrow, which is the whole reason a reliable way to collect and store the data you need is worth having before you need it.

I pulled all of this with a short script: read the Tranco list, fetch each robots.txt, parse the user-agent rules, count the blocks. If you want the same kind of web-scale snapshot for a question of your own, whether that is prices, listings, or who publishes what, that is the sort of thing we build at DataScrape Solutions.

Method, for anyone who wants to check it: Tranco top 1,000, fetched October 2, 2026. 485 of the 1,000 served a readable robots.txt; the rest were infrastructure domains, returned non-robots responses, or refused the request. "Block" means a full Disallow of the site root for that crawler's named user-agent.