Ran into an interesting edge case with pickle this week. I had a producer task that was querying objects from a database, and pickling them plus a reference to a callback function to pass to worker tasks. Everything was working fine, but I was getting sick of logging into a Django shell to invoke the workers with test data. So I wrote a quick
__main__ function in my task code to do the same thing.
To my surprise, this exact same code I had been typing into a shell was now throwing an exception when invoked from inside a
It turns out that when you pickle a function from inside a
__main__ block, the module reference that it will be pickled with is
__main__, not the actual module namespace. This is actually a known issue. In my case, it was easy to work-around; I simply put this code into a Django custom management command.
More discussion of this issue here.