Enable GhostFat auto-validation

[henrygab]

GhostFAT requires unit tests

GhostFAT is quite complex (file systems are inherently complex), and is critical functionality relied upon for updating devices. As seen with the RPi Pico, sometimes it even gets burned into ROM.

The need to validate GhostFAT is thus critical. However, today this is an extremely manual process.

For example, validation that it generates a correct file system:

• Build bootloader with all substantial variations (base, >1 sector per cluster, various sized flash files, etc.)
• Loading those bootloaders onto real devices
• Plus those devices into machines
• Dump the exposed USB flash drive to a disk image
• Manually validating the file system image is correct

Similar steps are needed to validate that GhostFAT will extract the relevant portions of a .UF2 file, and send them to the board at the correct offset.

In short, it's a huge manual test effort, and confidence drops as additional features / combinations are supported.

The automated validation should compile as board variants, to keep the build environment identical where possible.

Here's a straw man of features and test cases that should be part of automated CI checks:

Features:

• Verify generated file system is bit-for-bit match of known-good version (catch regressions)
  • Requires reproducible FS generation (explicit date/time stamp)
  • Requires compiling code native to build environment
  • Requires availability of known-good versions of file system
  • CI build to include new ports
  • CI build copies known-good file system to bin directory
  • CI build runs the native built test code after compilation

Test Cases:

• Basic Image (512-byte sector, 1 sector per cluster, FAT16, Flash size of 4MB @ offset zero)
• Various sectors per cluster (at least 8 and 64 sectors per cluster for 4k and 32k clusters)
• Various flash sizes and starting offsets (at least all those currently in use)
• Flash sizes that are not multiples of cluster size (only affects cases with >1 sector per cluster)
• Edge cases for compilation date / time stamps ? (e.g., Feb 29th)

For Hathach to add via Ceedling/CMock/Unity

• Verify written UF2 block result in writes to correct flash offsets, with expected data
• Verify written UF2 blocks for non-matching BOARD_UF2_FAMILY_ID are NOT written to flash

[henrygab]

See PR #94 for file system validation solution.

At time of writing, code is written that:

1. Defines a "port" called test_ghostfat
2. Defines various boards under that port which exercise different ghostfat configuration (clusters per sectors, flash size)
3. Compiles executable that is native to the build environment
4. Each build outputs a file "ghostfat.img"
5. Each build attempts to validate "ghostfat.img" against "knowngood.img"

In short, it's essentially written, except for the build integration and direct use of compressed known good image.

[hathach]

thanks @henrygab very comprehensive, I always admire your passionate and effort put into PRs. This is a great idea, we can also include this test in the CI to make sure we won't break it while refactoring. Normally I would use Ceedling/CMock/Unity for testing since mocking/faking hardware make it easier to test with. Though please continue with your great work, later on I will try to see if I could use part of them as unit test :)

[henrygab]

Mocking is insufficient for testing of such a complex item as file system generation. I will focus on this.

However, those seems like a very good fit for verifying the UF2 blocks "written" via ghostfat are properly translated into writes to the appropriate flash offset. I will leave this to you.

[henrygab]

I see the action exists, but does not appear to ever get executed. The point of these tests was to address the very high cost of manual validation of the file system generation ... by ensuring those same inputs give identical outputs ... and thus avoid unintentional changes in what's generated.

I imagine some changes have been intentional since this was written. If I get this action running again, would you consider running it weekly?

[hathach]

I think the current CI already do some test with before and after changes and then compare with known-good file system images. You make that brilliant PR, but I kind of forgot the details