Why isn't zope using `copyreg` extension registry ?

[perrinjerome]

Using copyreg extension registry seems a very easy way to reduce the size of pickles, there's even a range reserved to Zope:
https://github.com/python/cpython/blob/3.14/Lib/copyreg.py#L217

Is there a reason not to use it ?

I think this cannot be used for persistent classes, because only their state is pickled, but for non persistent classes such as DateTime.DateTime this looks like a very easy way to have smaller pickles.

[dataflake]

I would guess the only reason it has not been used is because no one thought this a priority or no one wanted to invest the time.

[perrinjerome]

Thanks for feedback @dataflake

I think this cannot be used for persistent classes, because only their state is pickled

that was wrong, it seems it can also be used for persistent classes.

For ERP5 we have a benchmark test which creates documents during approx 2 hours, I experimented with running this with a patch to register copyreg extensions for most common types, such as BTrees.OOBTree.OOBTree, Persistence.mapping.PersistentMapping and some application specific classes. The size of the ZODB file storage at the end of the this test went down from 2 456 Mo without copyreg extensions to 2 100 Mo with extensions, so it seems using copyreg can have a good impact on the disk usage.

I have been searching a bit in the zope mailing list archives, but I could not find discussion about using it in Zope. The only references I found is that it comes from PEP-307, it was discussed a bit on python-dev mailing list in this thread. This feature is not documented in the library reference, it seems it was never documented ( python2 doc https://docs.python.org/2.7/library/copy_reg.html did not mention it ).

So from what I see, the advantage of using this would be that disk usage can be significantly reduced, this quick test showed 15% gain. The downside is that once pickles are saved in a ZODB with some extensions registered, it's necessary to have the exact same extensions registered to be able to load the pickles - so if Zope starts using it, it's not possible to change our mind later.

My feeling is that it's an easy quick win for situations where having ZODB using less disk space is important, but I don't want to push for Zope to be using it by default. Maybe applications based on Zope or ZODB can decide to use copyreg extensions if they accept the fact that they have to keep using the same extensions forever.

the patch to register the extensions

import copyreg

# Zope classes
copyreg.add_extension('DateTime.DateTime', 'DateTime', 128)
copyreg.add_extension('BTrees.OOBTree', 'OOBTree', 129)
copyreg.add_extension('BTrees.OOBTree', 'OOTreeSet', 130)
copyreg.add_extension('BTrees.OOBTree', 'OOBucket', 131)
copyreg.add_extension('BTrees.Length', 'Length', 132)
copyreg.add_extension('BTrees.OIBTree', 'OIBucket', 133)
copyreg.add_extension('Persistence.mapping', 'PersistentMapping', 134)

# ERP5 classes
copyreg.add_extension('Products.ERP5Type.Workflow', 'WorkflowHistoryList', 192)
copyreg.add_extension('erp5.portal_type', 'Sale Packing List', 193)
copyreg.add_extension('erp5.portal_type', 'Sale Order Line', 194)
copyreg.add_extension('erp5.portal_type', 'Applied Rule', 195)
copyreg.add_extension('erp5.portal_type', 'Simulation Movement', 196)

[dataflake]

I am scared of the fact that none of this was ever documented. It feels like they all thought it was a good idea more than 20 years ago but when it was implemented no one actually used it.

Ask yourself, is this what you want to trust your production data to? I wouldn't, especially if the only gain is a small decrease in ZODB size.

[navytux]

@tim-one, any input from you on this maybe? You were the original author of that https://peps.python.org/pep-0307/#the-extension-registry and did a lot of work around ZODB and Zope in general. (I'd like to say thanks for that using the occassion).