Like CURRENT.UF2, could there be EVERYTHING.UF2 w/ entire flash filesystem? (code.py,libs,etc)

[todbot]

A Discord discussion led me to want the ability to have something like CURRENT.UF2, but contained both the current version of CircuitPython and the contents of the CP filesystem (code.py, lib, etc)

Would it even be possible for this hypothetical EVERYTHING.UF2 virtual file to be created? I know this is outside of the purview of the bootloader, but if the bootloader takes an entire snapshot of all known flashes and restores them, would it work?

[hathach]

it depends on the port, can you tell which board you are using ?

[todbot]

I was hoping it could be platform-independent. I use almost all the CircuitPython platforms pretty regularly: ESP32-S2, SAMD, and RP2040. (I understand RP2040 is ROM-based UF2)

[hathach]

Platform-independent still need port specific implementation to pull this off.

• SAMD is not supported by this repo, you should check out its implementation at https://github.com/adafruit/uf2-samdx1 and/or https://github.com/microsoft/uf2-samdx1
• ESP32S2 currently only support the actual firmware since its flash is partitioned, and the filesystem is placed on a separated parition. Having EVERYTHING.UF2 is possible, though current implementation limit the flash write to only 1 app partition to prevent error. If you really want it please open an issue specifically for esp32s2, though it is not on priority list and may take a while to implement
• RP2040 is bootrom you could raise issue here and hope that they could update it for the next chip revision https://github.com/raspberrypi/pico-bootrom

[T94T]

We are also interested in exact the same feature. Mainly on ESP32S2.

[henrygab]

For chips that use a partition, perhaps it would be possible to update to expose each of the partitions as UF2 binaries?

This would require:

1. ability for boards to indicate, at initialization time (uf2_init), at least:
   a. the number of partitions to expose
   b. the start and end address for each exposed partition
   c. optionally, an 8.3 fat-compliant filename for each exposed partition
2. update ghostfat to:
   a. use embedded-safe C++11
   b. rework major portions to dynamically calculate (at uf2_init) values, rather than current #define compile-time constants (but: check RAM increases OK?)
   c. rework how files are handled (current code presumes exactly one UF2 file exposed)
3. define "graceful" failure modes ... esp. for cases that are currently prevented by compile-time assertions, but will move to dynamic validation

---

Embedded-safe C++

Sometimes called "C++ as a better C", it restricts what features of C++ are used. For example, absolutely prohibited features include:

• exceptions
• run-time type informatino (RTTI)
• dynamic allocation (e.g., most of STL)

Usable features that provide value include:

• constexpr functions ... safer macros!
• scoped enums
• namespaces
• libraries that enable zero-runtime-overhead type-safety (imagine! FAT cluster numbers no longer causing off-by-two errors!)

---

This would definitely take a while to implement, because the current ghostfat code has many compile-time constants and behaviors that ensured via static_assert. Changing those would increase testing, and the need to code defensively (runtime-checks, instead of compile-time ... and how to fail gracefully?).

[hathach]

@henrygab thank you for your informative input, though, I think bootloader should just focus on updating the firmware anything that relates to firmware usages such as its file system should be done by firmware itself. For embedded C++, I don’t think we need to migrate to it since not all MCUs have C++ compiler for it and we actually don’t have to do complicated thing within bootloader. I agree with defensive coding style, but it is best to write as little code as possible.

[henrygab]

Ok, you're the boss! 👍

[tannewt]

I believe that SAMD's CURRENT.UF2 already includes the filesystem. nRF may as well. It is definitely a handy feature.

[hathach]

@tannewt SAMD & nRF52 CURRENT.UF2 includes all internal flash contents therefore also include internal file system e.g nvm if available but not the file system on external flash. The S2/S3 has different partitions for firmware, nvm, file system and they may not be continuous, You probably understand this better than me.

For S2/S3, currently tinyuf2 limit itself within the firmware(ota) partition to prevent overwriting to other partitions. Therefore the address in uf2 is starting from zero 0x00 as start of partition. However we could enhance this behavior and dump everything into CURRENT.UF2 (after partition table and 2nd stage bootloader), and when writing back we simply write over everything. To prevent bricking incompatible partition table and/or different flash size e.g 4 MB vs 8 MB we could use vendor + product ID for the family ID to make sure it can only be written to the same board. Let me know if you have any other suggestion.

Ok, you're the boss! +1

Nah, I wouldn't want bootloader to know too much of app knowledge and just want to write as less code as possible. Putting them all into CURRENT.UF2 is better way to go.

NOTE: currently max size is around 32MB with 1 cluster = 1 sector config, which limit the flash chip to less than 16MB. To support 16MB and more, we need to enhance ghostfat to run with 1 cluster = n sectors (n to be decided later).

[tannewt]

Very good points @hathach. It'd probably be best to chat with the Microsoft UF2 folks about how we designate UF2s for different uses (aka partitions.) We wouldn't want to copy off the nvm partition data and then copy it over an ota section.

[henrygab]

FYI:

ESP32 appears to be hard-coded to only allow writes to the ota1 partition.

Moreover, the offset stored in existing .uf2 files is a relative offset for that ota partition.

Ghostfat has no concept of which file is being written to, and IO can be re-ordered by the time the device sees it (e.g., no guarantee that sector 0 will be seen as written by device before sector 10). Therefore, you cannot simply insert a "switch to partition X" (or any stateful) command into the head of the .uf2 file, as it may be applied before the device sees writes that should have been affected by that stateful command.

In combination, these factors make it challenging to support writing more than a single target partition in ghostfat. However, if you simply want to expose additional .uf2 files to allowing reading of other partitions (such as a file system), that would be (still non-trivial, per my earlier answer), but within the realm of possibility. But without the ability to write it back, what would be the value?

---

expand for links to relevant portions of code

The target address is embedded in each sector of the .UF2 file. See targetAddr field in uf2.h:

tinyuf2/src/uf2.h

Lines 86 to 95 in 9104f7c

	typedef struct {
	// 32 byte header
	uint32_t magicStart0;
	uint32_t magicStart1;
	uint32_t flags;
	uint32_t targetAddr;
	uint32_t payloadSize;
	uint32_t blockNo;
	uint32_t numBlocks;
	uint32_t familyID;

Ghostfat provides the buffer provided by the .UF2 sector to the underlying board:

tinyuf2/src/ghostfat.c

Line 498 in 9104f7c

board_flash_write(bl->targetAddr, bl->data, bl->payloadSize);

As an example, the Espressif port caches stuff:

tinyuf2/ports/espressif/boards/board_flash.c

Lines 109 to 122 in 06a65c8

	void board_flash_write (uint32_t addr, void const *data, uint32_t len)
	{
	uint32_t new_addr = addr & ~(FLASH_CACHE_SIZE - 1);
	if ( new_addr != _fl_addr )
	{
	board_flash_flush();
	_fl_addr = new_addr;
	board_flash_read(new_addr, _fl_buf, FLASH_CACHE_SIZE);
	}
	memcpy(_fl_buf + (addr & (FLASH_CACHE_SIZE - 1)), data, len);
	}

But... the board_flash_read() and board_flash_write() functions are hard-coded to only allow writing to the ota0 partition:

tinyuf2/ports/espressif/boards/board_flash.c

Lines 72 to 75 in 06a65c8

	void board_flash_read(uint32_t addr, void* buffer, uint32_t len)
	{
	esp_partition_read(_part_ota0, addr, buffer, len);
	}

---

[hathach]

FYI:

ESP32 appears to be hard-coded to only allow writes to the ota1 partition.

Moreover, the offset stored in existing .uf2 files is a relative offset for that ota partition.

Yup, currently tinyuf2 for esp32s2/s3 is writing as offset to ota0 to prevent overflow to other partition.

In combination, these factors make it challenging to support writing more than a single target partition in ghostfat. However, if you simply want to expose additional .uf2 files to allowing reading of other partitions (such as a file system), that would be (still non-trivial, per my earlier answer), but within the realm of possibility. But without the ability to write it back, what would be the value?

I think the most wanted featured is not exposing additional partition as uf2. User want to take the snapshot of the current working state of application(firmware + data) and then clone/deploy/duplicate it to another board. Therefore I think packing all the flash together within a single all-in-one UF2 make more sense. However, since the current code use address as offset to 1st partition (start address = 0), I would suggest to use vendor+board id as family for the all-in-one uf2 in addition to current offset scheme.

[todbot]

Coming back to this after 3 years: having a single UF2 to install would make loading up all the code on the MEMENTO board a lot easier.

[SlayerPower]

ah, I wouldn't want bootloader to know too much of app knowledge and just want to write as less code as possible. Putting them all into CURRENT.UF2 is better way to go.

NOTE: currently max size is ar

[SlayerPower]

igual mente gracias . gracias por todo pe osti jajaja, o ye com2 estas