Unpicklingerror: Newobj Class Argument Isn't A Type Object

I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them, based on the code from Shane Hathaway here: Pyth

Solution 1:

Successful pickling happens in two steps, the pickle.dump by the Pickler, and the pickle.load by the Unpickler. The Pickler converts an object to a serialized format (e.g. strings), and the Unpickler digests the pickled object and generates a new object that should be equivalent to the original. Pickle has several functions that can be used to dump pickles... so part 1 is getting the objects to convert to the serialized format. With a custom pickler, you can bypass some of python's safeguards to pickle objects that pickle itself can't pickle. Following your example, I could create a simple Pickler, that would convert lambdas and whatnot to strings by converting each object to it's __repr__.

>>>x = lambda x:x>>>repr(x)
'<function <lambda> at 0x4d39cf0>'
>>>>>>import pickle>>>l = repr(x)>>>pickle.dumps(l)
"S'<function <lambda> at 0x4d39cf0>'\np0\n."

This would definitely be pickleable, as it's a string. However, the problem is how to build an object from the saved string. For a lambda, if you had a function that could look-up the memory reference noted in the string, you could get the object back... but only if you still had the original object alive in your memory... so that's no good. So the trick of converting to strings only works when there's enough information contained in the __repr__ string to build a new object from the stored string's information. You could get fancier about what you store, but you will eventually run into problems by converting the objects to strings, most likely. So this is a case where your Pickler would work, but your Unpickler would fail.

Dictionaries are interesting to pickle, because they can have anything in them, and usually do pretty quickly. One of the nastiest dictionaries is the globals() dictionary. To serialize it, I'd use dill, which can serialize almost anything in python.

>>> import dill
>>> >>> deffoo(a):
... defbar(x):
... return a*x
... return bar
... >>> classbaz(object):
... def__call__(self, a,x):
... return foo(a)(x)
... >>> b = baz()
>>> b(3,2)
6>>> c = baz.__call__
>>> c(b,3,2)
6>>> g = dill.loads(dill.dumps(globals()))
>>> g
{'dill': <module 'dill'from'/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/dill-0.2a.dev-py2.7.egg/dill/__init__.pyc'>, 'c': <unbound method baz.__call__>, 'b': <__main__.baz object at 0x4d61970>, 'g': {...}, '__builtins__': <module '__builtin__' (built-in)>, 'baz': <class'__main__.baz'>, '_version': '2', '__package__': None, '__name__': '__main__', 'foo': <function foo at 0x4d39d30>, '__doc__': None}

Actually, dill registers it's types into the pickle registry, so if you have some black box code that uses pickle and you can't really edit it, then just importing dill can magically make it work without monkeypatching the 3rd party code.

Or, if you want the whole interpreter session sent over as an "python image", dill can do that too.

>>># continuing from above>>>dill.dump_session('foobar.pkl')>>>>>>^D
dude@sakurai>$ python
Python 2.7.5 (default, Sep 30 2013, 20:15:49) 
[GCC 4.2.1 (Apple Inc. build 5566)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>import dill>>>dill.load_session('foobar.pkl')>>>c(b,3,2)
6

Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.