Some Websites Block Selenium Webdriver, How Does This Work?

So I'm trying to web crawl clothing websites to build a list of great deals/products to look out for, but I notice that some of the websites that I try to load, don't. How are webs

Solution 1:

Selenium uses a real web browser (typically Firefox or Chrome) to make its requests, so the website probably has no idea that you're using Selenium behind the scenes.

If the website is blocking you, it's probably because of your usage patterns (i.e. you're clogging up their web server by making 1000 requests every minute. That's rude. Don't do that!)

One exception would be if you're using Selenium in "headless" mode with the HtmlUnitDriver. The website can detect that.

Solution 2:

It's very likely that the website is blocking you due to your AWS IP. Not only that tells the website that somebody is likely programmatically scraping them, but most websites have a limited number of queries they will accept from any 1 IP address. You most likely need a proxy service to pipe your requests through.