Python Pycryptodome Aes-gcm Encryption Code Performance Improvement

I am having around 19G of data which I am doing tar and then encrypt. I use below code to do the job. from subprocess import call from Crypto.Cipher import AES from Crypto.Random i

Solution 1:

GCM is hard (though not impossible) to parallelize. Still, on my 3-year x86 laptop (with AESNI and CLMUL accelerated instructions) I do get 150 MB/s with PyCryptodome's GCM. That is only 2 minutes for 19GB, not a day! I used the following toy code:

data = os.urandom(1024*1024)
cipher = AES.new(key, AES.MODE_GCM)
for _ in range(1024):
    cipher.encrypt(data)
tag = cipher.digest()

The code is not directly usable for your use case, but it indicates that there might be an issue with you encrypting the full 19GB at once. Perhaps, you should instead break up the processing in chunks.

Some other comments:

• Use a profiler to identify where your program takes the most time. It might not be where you think it is (e.g. what about the tar step?).
• Ensure you are using the latest version of PyCryptodome (3.6.6), since CLMUL acceleration was added only recently.
• GCM can only encrypt 256GB at most - you are not that far from that with 19GB.