Which US Retailers Are Hardest to Scrape? We Tested the Top 14

If you point a plain script at a retail site and it comes back with a 403, that tells you almost nothing. A default HTTP client gets blocked on easy sites too, because its TLS handshake, the fingerprint it presents before a single byte of the page loads, marks it as "not a browser" instantly. So "my scraper got blocked" is not a measure of how hard a site is. It is a measure of how obvious your scraper is.

To find out what actually makes a US retailer hard to scrape, you have to separate the things that get you blocked: your IP, your TLS fingerprint, and whether you run a real browser. So we tested the 14 biggest US retail sites four different ways and watched which gate each site really enforces. The results are not what the "big name means hard" assumption would tell you.

How we tested it

We ran each of the 14 sites through four setups, changing one variable at a time.

One, a plain scripted request from a clean residential connection. Two, the same scripted request but with a real Chrome TLS fingerprint, using curl_cffi, which impersonates a browser's handshake, still a script and not a browser. Three, a real browser running on a datacenter IP. Four, the same real browser on a residential IP.

For each site we read the response headers and cookies to identify the anti-bot vendor, Akamai Bot Manager, DataDome, PerimeterX, Cloudflare, Imperva, by the fingerprints they leave behind (cookies like _abck and bm_sz for Akamai, datadome for DataDome, _px for PerimeterX). And we checked the actual page content, not just the status code, so a long "please verify you are human" page does not get counted as a success. Comparing those four results isolates exactly which gate stops you at each site.

What actually blocks you: three separate gates

Across all 14 sites, the blocking came down to three independent gates. A site can enforce one, two, or all three, and you have to pass every one it uses.

Gate one is IP reputation. The exact same real browser loaded Etsy, Wayfair, Chewy, Macy's and Kroger with no problem from a residential IP, and got blocked outright from a datacenter IP. Same browser, same everything, only the IP changed. Datacenter IPs are effectively dead for retail scraping, which is the entire reason residential proxies exist.

Gate two is your TLS fingerprint. From the same residential IP, swapping a plain request for one with a real Chrome TLS handshake unlocked Home Depot, Nike and Macy's, no browser, no new proxies, just a handshake that looked like Chrome. The fingerprint was the only thing that had been giving the scraper away. This is the cheapest, highest-leverage fix most people miss.

Gate three is browser execution. Etsy (DataDome) and Wayfair (PerimeterX) blocked even a script with a perfect Chrome TLS fingerprint on a clean residential IP. They run JavaScript challenges and behavioral checks that a script cannot answer. Only a real browser got through. So for those sites, a good IP and good TLS are necessary but not enough.

The full results

Both scripted tests ran from a residential IP. "Loads" means the real page came back; "Blocked" means a 403, 429, or challenge page; "Geo-gate" means a country-selection interstitial instead of the store.

The three tiers of US retail

Group the 14 sites by the hardest gate each one enforces, and a clear ranking appears.

Easy, where a residential IP plus a browser-grade TLS fingerprint was enough and no headless browser was needed: Costco, Walmart, Target, Lowe's, Home Depot, Nike and Macy's all returned full, real pages to a simple script once it had a Chrome fingerprint. Costco was the softest of all, running essentially no bot management. The surprise is Home Depot, which stacks three vendors (Akamai, PerimeterX and Forter) and still folded to good TLS from a clean IP.

Hard, where you need a real browser and a perfect-TLS script is not enough: eBay, Chewy, Kroger, Etsy and Wayfair. All five blocked the script but loaded in a real residential browser.

Special cases: Amazon throws its own "continue shopping" interstitial at almost everything, including a real browser, so it is aggressive but inconsistent rather than a hard wall. Best Buy showed a country-selection interstitial to our script and datacenter tests, but loaded the full US store in a real browser on a residential connection, so its wall leans on the same signals as the rest rather than a hard geo-block.

One pattern across all of it: Akamai Bot Manager is the most common vendor, on 9 of the 14 sites, but on most of them it blocks only the obvious signals, a bad TLS fingerprint or a datacenter IP. The vendors that demanded a full browser even with good TLS were DataDome (Etsy) and PerimeterX (Wayfair), plus a few stricter Akamai setups (eBay, Chewy, Kroger).

What this means if you are scraping retail

The working setup for retail at scale is all three gates at once: a residential IP, a browser-grade TLS fingerprint, and a real headless browser for the strict sites. Drop any one and a chunk of retail goes dark. This is exactly why a cheap datacenter-proxy scraper comes back with blanks on half the sites, and why "it worked on my laptop" quietly breaks in production. Your laptop had a residential IP and a real browser. Your server has neither.

It also means the difficulty is not really about the retailer. It is about how many of the three gates you are equipped to pass. A cheap self-serve tool passes zero of them well. A serious setup passes all three, and keeps passing them as the sites escalate.

That last part, keeping it working as sites change, is the real job. If you would rather have clean retail price or catalog data without building and babysitting that stack, that is what we do at DataScrape Solutions for clients. We handle the IP, the fingerprint, and the browser layer, maintain it as the retailers tighten their defenses, and hand over clean, validated data. You can start with a free sample of your exact targets so you see the data first.

FAQ

What is the hardest US retail site to scrape? In our test, the sites that blocked even a real browser on a datacenter IP and needed a full residential browser were Etsy (DataDome), Wayfair (PerimeterX), eBay, Chewy and Kroger. Amazon is also aggressive in a different way, showing its own interstitial to almost every automated request. The easiest was Costco, which runs almost no bot management.

Why does my scraper get blocked even with a good proxy? A clean IP is only one of three gates. Most scrapers get caught on their TLS fingerprint, the handshake a default HTTP client sends looks nothing like a browser, so anti-bot systems flag it before the page even loads. In our test, fixing only the TLS fingerprint (with a tool like curl_cffi) unlocked several sites from the same IP. And the strictest sites, DataDome and PerimeterX, need a real browser on top of that.

Do I need residential proxies to scrape retail sites? For the strict ones, yes. The identical browser that loaded Etsy, Wayfair, Chewy, Macy's and Kroger from a residential IP was blocked outright from a datacenter IP. Datacenter proxies are effectively dead for most retail. Easier sites like Costco or Walmart tolerate more, but a residential IP plus a browser-grade TLS fingerprint is the baseline that actually works.

Can you just deliver the retail data instead of me building this? Yes. We run the full stack, residential IPs, browser-grade fingerprints, and a real browser layer for the strict sites, maintain it as the retailers change their defenses, and hand you clean, validated data on your schedule. You can start with a free sample of your exact targets to see the quality before committing.