Python Requests Encoding Post Data

Version: Python 2.7.3 Other libraries: Python-Requests 1.2.3, jinja2 (2.6) I have a script that submits data to a forum and the problem is that non-ascii characters appear as garba

Solution 1:

Your client behaves as it should e.g. running nc -l 8888 as a server and making a request:

import requests

requests.post('http://localhost:8888', data={u'post': u'Andr\xe9 T\xe9chin\xe9'})

shows:

POST / HTTP/1.1
Host: localhost:8888Content-Length: 33Content-Type: application/x-www-form-urlencodedAccept-Encoding: gzip, deflate, compressAccept: */*User-Agent: python-requests/1.2.3 CPython/2.7.3

post=Andr%C3%A9+T%C3%A9chin%C3%A9

You can check that it is correct:

>>> import urllib
>>> urllib.unquote_plus(b"Andr%C3%A9+T%C3%A9chin%C3%A9").decode('utf-8')
u'Andr\xe9 T\xe9chin\xe9'

• check the server decodes the request correctly. You could try to specify the charset:

  headers = {"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8"}
  

  the body contains only ascii characters so it shouldn't hurt and the correct server would ignore any parameters for x-www-form-urlencoded type anyway. Look for gory details in URL-encoded form data

• check the issue is not a display artefact i.e., the value is correct but it displays incorrectly

Solution 2:

Try to decode into utf8:

unicode(my_string_variable, "utf8")

or decode and encode:

sometext = gettextfromsomewhere().decode('utf-8')
env = jinja2.Environment(loader=jinja2.PackageLoader('jinjaapplication', 'templates'))
template = env.get_template('mypage.html')
print template.render( sometext = sometext ).encode('utf-8')