Python Web Scraping [error 10060]

I am struggling to get my code, that scrapes HTML table info from web, to work through a list of websites held in ShipURL.txt file. The code reads in the web page addresses from Sh

Solution 1:

I would wrap your urlopen call with a try/catch. Like this:

try:
  shipPage = urlopen(shipUrl)
except Error as e:
  print e

That'll at least help you figure out where the error is happening. Without the extra files, it'd be hard to troubleshoot, otherwise.

Python errors documentation

Solution 2:

Websites protect their-selves against DDOS attacks by preventing successive access from a single IP.

You should put a sleep time between each access ,or at each 10 accesses or 20 or 50.

Or you may have to anonymize your access through tor network or any alternative

Solution 3:

Found some great info on this link: How to retry after exception in python? It is basically my connection problem so I decided to try until it succeeds. At the moment it is working. Solved the problem with this code:

while True:
                try:
                    shipPage = urllib2.urlopen(shipUrl,timeout=5)
                except Exception as e:
                    continuebreak

But I do thank everybody here, you helped me understand the problem a lot better!