Is There A Maximum Size For Python's Shelve Module?

I'm getting this exception when trying to open shelve persisted files over a certain size which is actually pretty small (< 1MB) but I'm not sure where the exactly number is. No

Solution 1:

Regarding error itself:

The idea circulating on the web is the data size exceeded the largest integer possible on that machine (the largest 32 bit (signed) integer is 2 147 483 647), interpreted as a negative size by Python.

Your code is running with 2.7.3, so may be a fixed bug.

Solution 2:

The code "works" if I change the depth from 2 to 1, or if I run under python 3 (after fixing the print statements and using items() instead of iteritems()). However, the list of keys is clearly not the set of keys found while iterating over the return value of recursive_dict().

The following restriction from the shelve documentation may apply (emphases mine):

The choice of which database package will be used (such as dbm, gdbm or bsddb) depends on which interface is available. Therefore it is not safe to open the database directly using dbm. The database is also (unfortunately) subject to the limitations of dbm, if it is used — this means that (the pickled representation of) the objects stored in the database should be fairly small, and in rare cases key collisions may cause the database to refuse updates.