Skip to content Skip to sidebar Skip to footer
Home / Connection / Network Programming / Proxy / Python / Python Requests

Requests - Proxies Dictionary

Post a Comment
I'm little confused about requests module, especially proxies. From documentation: PROXIES Dictionary mapping protocol to the URL of the proxy (e.g. {‘http’: ‘foo.bar:312

Solution 1:

Using the proxies parameter is limited by the very nature of a python dictionary (i.e. each key must be unique).

import requests

url = 'http://google.com'
proxies = {'https': '84.22.41.1:3128',
           'http': '185.26.183.14:80',
           'http': '178.33.230.114:3128'}

if __name__ == '__main__':
    print url
    print proxies
    response = requests.get(url, proxies=proxies)
    if response.status_code == 200:
        print response.text
    else:
        print'Response ERROR', response.status_code

outputs

http://google.com
{'http': '178.33.230.114:3128', 'https': '84.22.41.1:3128'}
<!doctype html><htmlitemscope=""itemtype="http://schema.org/WebPage"lang="en"><head><metacontent="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for."...morehtml...

As you can see, the value of the http protocol key in the proxies dictionary corresponds to the last encountered in its assignment (i.e. 178.33.230.114:3128). Try swapping the http entries around.

So, the answer is no, you cannot specify multiple proxies for the same protocol using a simple dictionary.

I have tried using an iterable as a value, which would make sense to me

proxies = {'https': '84.22.41.1:3128',
           'http': ('178.33.230.114:3128', '185.26.183.14:80', )}

but with no luck, it produces an error

Solution 2:

Well, actually you can, I've done this with a few lines of code and it works pretty well.

import requests


classClient:

    def__init__(self):
        self._session = requests.Session()
        self.proxies = Nonedefset_proxy_pool(self, proxies, auth=None, https=True):
        """Randomly choose a proxy for every GET/POST request        
        :param proxies: list of proxies, like ["ip1:port1", "ip2:port2"]
        :param auth: if proxy needs auth
        :param https: default is True, pass False if you don't need https proxy
        """from random import choice

        if https:
            self.proxies = [{'http': p, 'https': p} for p in proxies]
        else:
            self.proxies = [{'http': p} for p in proxies]

        defget_with_random_proxy(url, **kwargs):
            proxy = choice(self.proxies)
            kwargs['proxies'] = proxy
            if auth:
                kwargs['auth'] = auth
            return self._session.original_get(url, **kwargs)

        defpost_with_random_proxy(url, *args, **kwargs):
            proxy = choice(self.proxies)
            kwargs['proxies'] = proxy
            if auth:
                kwargs['auth'] = auth
            return self._session.original_post(url, *args, **kwargs)

        self._session.original_get = self._session.get
        self._session.get = get_with_random_proxy
        self._session.original_post = self._session.post
        self._session.post = post_with_random_proxy

    defremove_proxy_pool(self):
        self.proxies = None
        self._session.get = self._session.original_get
        self._session.post = self._session.original_post
        del self._session.original_get
        del self._session.original_post

    # You can define whatever operations using self._session

I use it like this:

client = Client()
client.set_proxy_pool(['112.25.41.136', '180.97.29.57'])

It's simple, but actually works for me.

Post a Comment for "Requests - Proxies Dictionary"