mask+state_mask leaving zero pixels raises exception

[NPounder]

If all pixels within the state mask are masked (e.g. cloud, QC masks) for a particular observation then KaFKA crashes with a ValueError. @GerardoLopez and I propose that is case should be prevented by the data handler so we do not propose to handle it within KaFKA. Instead, a commit in #16 Now submits an error to the logfile if this situation does occur during run time.

[TonioF]

That's possible if we also pass the state mask to the observations, so the two masks can be combined before they are handed out to kafka.

An important question that we should answer beforehand is: How will kafka deal with the case that data is available for some bands from one product, but not for others? This is a rare case, I know, but it can happen.

[NPounder]

Regarding the masks: KaFKa will still require both masks separately as the state mask indicates where the retrieval should be performed (and is the same for all images) while the observation mask changes how the retrieval is performed for each pixel in each image.

Regarding multiple products: I think that KaFKA should receive all the data that is available. Within a time step it will loop over all of the available data products in that time step and won't care if one time step has two different product types and the next has only one of them. It can already cope with multiple observations per time step. Handling multiple product types hasn't been implemented yet. @GerardoLopez has suggested that within a single product we should exclude any images where one of more of the bands are masked out - but that can be discussed.

Handling the double-masked images can be done in KaFKA if we decide that is the best place to do it. Perhaps this should be pre-processing rather than data-access.

[jgomezdans]

That's possible if we also pass the state mask to the observations, so the two masks can be combined before they are handed out to kafka.

This is something that kafka already has access to. We need both independently.

An important question that we should answer beforehand is: How will kafka deal with the case that data is available for some bands from one product, but not for others? This is a rare case, I know, but it can happen.

We just ignore the band with no data. This can be done fairly easily in kafka (ie not changing anything and not involving anything else in the chain), but can't ssh so can't test fix at the moment.

Although it might appear as a rare occurrence ~25% of MODIS data suffer from that, although other than saturation issues over bright snow/deserts, it's unlikely to happen on LC8/S2.

[TonioF]

We can only safely say whether all pixels are masked out after we have reprojected it to the correct grid, so I still think this check should go into the observations class.