External storage for images

[gianpispi]

Looking ad SwiftData, it would be great to have some externalStorage attribute for image fields, that automatically and atomically move the image to the documents folder and insert the path to the db.
This could be an even bigger impact for this to work with CloudKit out of the box - I saw it works already with blob fields.

I just don't like to store big images inside the database itself

[mbrandonw]

Hi @gianpispi, we have plans to look into this, but can't commit on a timeline. Referencing files on disk from the database brings extra complications, and I'm not sure even SwiftData handles it correctly. For example, do you know what SwiftData does if one deletes the file from disk directly? Does SwiftData observe that and clean up the data in the database?

[aehlke]

The pattern I like from prior Realm<->CloudKit libraries is to store Data blobs on disk and use CKAsset for them automatically. The files are stored somewhere inaccessible to the user alongside the DB files so they can be expected to exist, with some kind of edge case error handling.

[mbrandonw]

That all sounds good in theory, but still comes with many complications.

For example, SwiftData works how you are describing, where you can use .externalStorage to say that some data should be stored on disk instead of in the DB. But then you can write a reasonable looking query, like #Predicate { $0.image.count > 1_000 } and it will actually crash at runtime because the data isn't really stored in the db, it's on disk. Even worse, small blobs of data are stored in the DB, and so that query could sometimes work and sometimes crash, giving you a false sense that you know how the system works.

All of this to say, we'd love to do something about this, but it really is not as easy as it seems at first. We think it is a better model to hold onto actual file URL's in the model rather than pretending that the data is held in the database.

[aehlke]

I see. As a Realm refugee I'm still learning about this tool.

An incremental improvement you can do is to use CKAsset for Data/blobs even if SQLite still stores the data

[mbrandonw]

An incremental improvement you can do is to use CKAsset for Data/blobs even if SQLite still stores the data

That is already supported: https://github.com/pointfreeco/sharing-grdb/blob/05c78647b5fdf5ae8b274da6f7f2934708963f00/Sources/SharingGRDBCore/Documentation.docc/Articles/CloudKit.md#Assets

[lhr0909]

I was able to manage these externally by syncing them myself onto CloudKit. I create separate CKRecord type and set up the asset field. When a new record is created from the main table, I will make sure the asset gets uploaded as a separate CKRecord, then I store the recordName as an ID field (something like assetCKRecordID), then when loading I will look for the ID, and if it exists and the file URL is not, I will fetch from CloudKit. Hope this helps. It would be great if we can have a hook for the sync (since we already see debug logs to know when the sync happens), so we can hook onto them from the delegate and manually manage these external assets. Feels safe enough to do (for now).

[mbrandonw]

Hi @lhr0909, we'd definitely be down to opening up more delegate methods to make things like this easier to accomplish outside of the library. However, where are you storing these new CKRecords? Is it in the same CKContainer that your SyncEngine is running off of?

I ask because anytime the sync engine sees a record type it doesn't understand it caches the record ID because it will need to try the record again in the future if the database schema updates. And so that means the sync engine will indefinitely cache the record IDs of these records you are creating outside the purview of the sync engine.

If this is a common pattern (people storing records in their container that SyncEngine doesn't know about) then we may need to allow the engine to be initialized with a list of record types to ignore.

[lhr0909]

@mbrandonw that's correct, I am currently using the same CKContainer for this. Is it advised to use a separate CKContainer so SyncEngine caches less information? What would happen if the sync engine keeps caching these exotic record IDs?

I personally also prefer to have the assets in the file system because it feels like we could load less data into memory when we use the files instead of loading blobs from db in the app. And looking at the CKAsset init and syncing methods, it seems like it uses file system to hold information as well, so it definitely feels more natural to me if SQLiteData can just convert a URL column into CKAssets and help manage them.

This is one of the blockers for me to enable CloudKit in my bigger GRDB-powered app (I already hold half a gigabyte worth of files when I use my own app, and I am expecting some of my users to have more), but I think the externally managed way I just built out will work fine for now. I could open a separate CKContainer for the assets for sure.

[mbrandonw]

@mbrandonw that's correct, I am currently using the same CKContainer for this. Is it advised to use a separate CKContainer so SyncEngine caches less information?

It's just not something we've given much thought to yet. The sync engine definitely assumes right now any records created in the container came from its SQLite database, but we didn't consciously make that decision and it doesn't necessarily have to be that way. It seems like a pain to have to create a whole new container just to store additional records that are not in the SQLite database

What would happen if the sync engine keeps caching these exotic record IDs?

The table will just continue to grow, theoretically creating more work for it whenever it detects a schema change and needs to re-process the records. However, those records get cleared out after a schema change, so I suppose they won't grow indefinitely.

This is one of the blockers for me to enable CloudKit in my bigger GRDB-powered app (I already hold half a gigabyte worth of files when I use my own app, and I am expecting some of my users to have more), but I think the externally managed way I just built out will work fine for now. I could open a separate CKContainer for the assets for sure.

Yeah, if your app is expected to have gigabytes of assets and is very asset-heavy, then it is probably not appropriate to store them in SQLite.

I think ideally we would support this as a first class feature of SyncEngine, but until then we certainly can try to make things a little nicer for people. We'll chat a bit more about it and share our thoughts soon.

[mbrandonw]

The table will just continue to grow, theoretically creating more work for it whenever it detects a schema change and needs to re-process the records. However, those records get cleared out after a schema change, so I suppose they won't grow indefinitely.

I misspoke here a little. The records will be ignored if their record ID does not fit the format "{id}:{type}" because that's the convention SQLiteData uses. So for the most part records you create will just silently be ignored by SQLiteData.

There is still a little bit of upfront work the sync engine will do before it gets to the point of ignoring those records, and so we may be able to optimize things a bit by allowing you to specify record types to fully ignore.