Namedtemporaryfile Speed Underwhelming

Im trying to use a NamedTemporaryFile and pass this object to an external program to use, before collecting the output using Popen. My hope was that this would be quicker than crea

Solution 1:

I will try to answer your question although I am not very experienced with Python runtime efficiency.

Drilling in the code of Python's tempfile.py you can find a clue about what might take some time. The _mkstemp_inner function might open a few files and raise an exception for each one. The more temp files your directory contains, the more file name collisions you get, the longer this takes. Try to empty your temp directory.

def_mkstemp_inner(dir, pre, suf, flags):
    """Code common to mkstemp, TemporaryFile, and NamedTemporaryFile."""

    names = _get_candidate_names()

    for seq inrange(TMP_MAX):
        name = next(names)
        file = _os.path.join(dir, pre + name + suf)
        try:
            fd = _os.open(file, flags, 0o600)
            _set_cloexec(fd)
            return (fd, _os.path.abspath(file))
        except OSError as e:
            if e.errno == _errno.EEXIST:
                continue# try againraiseraise IOError(_errno.EEXIST, "No usable temporary file name found")

Hope that helped.