What Is The Fastest Way In Python To Convert A String With Formatted Numbers In An Numpy Array

I have a large ASCII file (~100GB) which consists of roughly 1.000.000 lines of known formatted numbers which I try to process with python. The file is too large to read in complet

Solution 1:

Have you tried numpyp.fromstring?

np.fromstring(line, dtype=np.float, sep=" ")

Solution 2:

The np.genfromtxt function is a speed champion if you can get it to match you input format.

If not, then you may already be using the fastest method. Your line-by-line split-into-array approach exactly matches the SciPy Cookbook examples.