Importing Csv Embedding Special Character With Numpy Genfromtxt

I have a CSV containing special characters. Some cells are arithmetic operations (like '(10/2)'). I would like to import these cells as string in numpy by using np.genfromtxt. What

Solution 1:

Looks like the file may have the 'other' divide symbol, the one we learn in grade school:

In [185]: b'\xc3\xb7'
Out[185]: b'\xc3\xb7'
In [186]: _.decode()
Out[186]: '÷'

---

Recent numpy version(s) handle encoding better. Earlier ones tried to work entirely in bytestring mode (for Py3) to be compatible with Py2. But now it takes an encoding parameter.

In [68]: txt = '''(10/2), 1, 2
    ...: (10/2), 3,4'''

In [70]: np.genfromtxt(txt.splitlines(), dtype=None, delimiter=',')
/usr/local/bin/ipython3:1: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use Nonefor the system default.
  #!/usr/bin/python3
Out[70]: 
array([(b'(10/2)', 1, 2), (b'(10/2)', 3, 4)],
      dtype=[('f0', 'S6'), ('f1', '<i8'), ('f2', '<i8')])

In [71]: np.genfromtxt(txt.splitlines(), dtype=None, delimiter=',',encoding=None
    ...: )
Out[71]: 
array([('(10/2)', 1, 2), ('(10/2)', 3, 4)],
      dtype=[('f0', '<U6'), ('f1', '<i8'), ('f2', '<i8')])

Admittedly this simulated load from a list of strings is not the same as loading from a file. I don't have earlier numpys installed (and not on Py2), so can't show what happened before. But my gut feeling is that "(10/2)" shouldn't have given problems before, at least not in an ASCII file. There aren't any special characters in the string.

---

With the other divide:

In [192]: txt ='''(10÷2), 1, 2
     ...: (10÷2), 3,4'''In [194]: np.genfromtxt(txt.splitlines(), dtype=None, delimiter=',',encoding='ut
     ...: f8')
Out[194]: 
array([('(10÷2)', 1, 2), ('(10÷2)', 3, 4)],
      dtype=[('f0', '<U6'), ('f1', '<i8'), ('f2', '<i8')])

Same thing in a file:

In [200]: np.genfromtxt('stack49859957.txt', dtype=None, delimiter=',')
/usr/local/bin/ipython3:1: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use Nonefor the system default.
  #!/usr/bin/python3
Out[200]: 
array([(b'(10\xf72)', 1, 2), (b'(10\xf72)', 3, 4)],
      dtype=[('f0', 'S6'), ('f1', '<i8'), ('f2', '<i8')])

In [199]: np.genfromtxt('stack49859957.txt', dtype=None, delimiter=',',encoding=
     ...: 'utf8')
Out[199]: 
array([('(10÷2)', 1, 2), ('(10÷2)', 3, 4)],
      dtype=[('f0', '<U6'), ('f1', '<i8'), ('f2', '<i8')])

In earlier versions, encoding could be implemented in a converter. I've helped with that task in previous SO questions.