Add the blakey generator based on BLAKE3

[sbp]

This PR adds a new generator called blakey, based on BLAKE3. It deprecates the old dkim generator.

The new generator is fast and produces short output, at just 32 characters. It uses a deterministic, cryptographically secure hash of the conjunction of the lid and the raw message source received by archiver.py, with 160 bits of security. The encoding is the same as the dkim generator, which means it avoids the likelihood of taboo substrings.

The main difference between the new and old generators is that blakey does not attempt to deduplicate inputs. In other words, whereas the dkim generator was intended to simulate a DKIM signature session input, giving the same output for different message sources which were nevertheless regarded as equivalent according to the algorithm, the blakey generator is more like the full generator in that it generates a different output for every lid and message source pair.

The dkim generator was an early version of the one developed and discussed in PR 517 of Ponymail. That PR generated considerable disagreement as to the best method for judging two messages to be equivalent and encoding the result. There were several underlying problems, but the main one was the tension between wanting to deduplicate as thoroughly as possible by dropping information, and wanting to provide pristine archives by retaining information.

The blakey generator sidesteps this issue by retaining all information. The only benefit it shares in common with dkim is that the output is short and therefore better suited for shareable permalinks.

Since all information is retained in the new generator, naturally this means that input messages are no longer deduplicated. If this PR is accepted, the idea is that deduplication could instead occur as a separate process implemented either in archiver.py or in a background task. When a new message is received, existing messages could be checked for similarity. If a similar enough existing message is found, the new message can either be stored in the archive with a reference to the existing message, or discarded. Such a check would be quite efficient because, for example, the dkim generator considered messages with the same message-id to be equivalent, and a message-id query would dramatically narrow down the candidates to check for similarity. The interface can then redirect or ignore requests for messages that have a reference to an existing message.

This PR is, therefore, the first part in what could be a sequence of updates to Foal. This PR introduces blakey, and deprecates dkim. Next it would be useful to add a tool which would migrate databases using dkim to use blakey instead. Then various deduplication tools could be added, to archiver.py or as a background process, or both.

The significant advantage of this approach is that when deduplication takes place as a separate task with aliasing, it can not only be performed in a non-destructive way but it can also be flexible. The approach can be changed on the fly, and existing emails are unaffected. It does not require there to be a consensus on the difficult problem of what emails are equivalent. And it allows such approaches to be added incrementally.

[sbp]

The build is failing because the PyPI blake3 module requires Rust and Cargo. This PR requires either the blake3 module to be installed, or Python 3.10. Unfortunately since Python 3.10 isn't released yet that means for testing we'd need to add a dependency on Rust and Cargo. Alternatively the PR could be redone with SHAKE-256.

[sebbASF]

What is the advantage of this generator?
AFAICT, it makes preservation of Permalinks much harder.
Unless one has the exact original input, it's not possible to rebuild a database using the same links.
For example, importing mails from mbox-vm would result in different links than importing from mod_mbox, and a different link again would result from importing from ezmlm archives, or from any other source.

[Humbedooh]

I am not in favor of this PR as it stands.
I don't think it should touch the DKIM generator, I am quite fond of that one and would prefer it stay as is.
DKIM and BLAKE3 solve two distinctly different problems, so I'd expect this to be a current alternative and not a replacement.

[sbp]

There two sets of problems with DKIM-ID:

(1) The message sent by the poster, the message delivered to the mailing list manager (MLM), the message archived, and the message received by mailing list subscribers may all have different DKIM-IDs to one another, especially if Received and Delivered-To were included in the DKIM-ID algorithm. Most MLMs break DKIM by default, which RFC 6377 was written to address, and it may not even be possible to configure some of them to not do that. Moreover, archiver.py can be arbitrarily set up to capture any of these potentially differing emails.

(2) There is no agreement on the DKIM-ID algorithm. Should the lid be in the input, or appended to the DKIM-ID always, or appended to the DKIM-ID conditionally (a question that was attempted to be resolved twice with no response)? Should the Received and Delivered-To headers be included? What if new essential headers are added to email in future, such as the ARC headers? What if a site manager wants to change the level of deduplication? As was said in the previous PR thread:

I think it's vital that we do not implement the a new generator until we are sure that all issues have been addressed.
Every change to the algorithm reduces Permalink stability.

Also, Foal currently uses an outdated version of the DKIM-ID proposal.

[sebbASF]

AFAICT the blakey generator does not even attempt to deal with the issue of issuing a stable ID for equivalent emails, so comparisons on that basis are irrelevant.

[sbp]

The purpose of this PR was to fix the issues newly documented as #44 and #45. Potential solutions different from the ones suggested by this PR, and hopefully more appealing, have been suggested in the new issue submissions. This PR is therefore deprecated.