Modifying Endian output of products

[houchin]

Hi,

I have modified the config in my ADL installation to output only big endian VIIRS verified RDR files (and set it to DMS to actually get the files). When I run that portion of the VIIRS controller, it produces a <URID>.VIIRS-SCIENCE-RDR-Verified.BE file as expected, and the .asc file indicates big endian.

However, when I then go to try to use that file in a second configuration of the controller set to run only the ProSdrViirsCal algorithm, I get the following error:

Code: Select all

DBG_HIGH DmApiClient.cpp|548| tid-47916145926352 Error restoring the following saveset (referencing the URID of the input Verified RDR)

What does this mean? Even with DBG_LOW, I don't get any more information.

I'm running on little endian Linux.

EDIT: I changed the configuration back to output both endians and I still get the same error, this time on the native/little endian version of the file.
EDIT 2: This is with NCT 3 and NCT 4 input VIIRS data. I'm getting the same error on both my Aerospace ADL install and on GRAVITE.

FYI: I configured ADL to output only big endian files because I don't like having both variations around, especially in that sometimes we're working without metadata. The cost to byteswap the file is minimal.

[kbisanz]

There's (obviously) some problem with restoring the Verified RDR item you've previously produced. I've been able to replicate the problem, but haven't determined a cause or solution yet.

[kbisanz]

The problem is that when verified RDRs are created, their ObservedDateTime metadata is not set because it is not known. So, it's defaulted to the IET epoch time, which is January 1, 1958. This causes problems because it's earlier than the UNIX epoch time of January 1, 1970. I believe we have a fix for it, but it's still being tested.

I'm sure you're already doing this, but as a work around you can run the verified RDR algorithm as part of the controller so that it doesn't have to read the offending item in from disk (it'll still be in memory). You'll need to remove the offending item from your input directory.

I just wanted to let you know we haven't forgotten about this issue.

[houchin]

Thanks for the update. I've been on vacation for a few days, so nothing has happened with this on my end.

What I want to to is to create a configuration where the verified RDR and geolocation algorithms would be run through the VIIRS SDR controller, with output after geolocation. Our analysts could then run multiple configurations of the VIIRS SDR calibration algorithm on that input data, preventing geolocation from being run over and over. My assumption was that I needed to run them in strict order. However, would it be better to reset the verified RDR configuration to HEAP and then have the controller always run verified RDR (skipping over gelocation in this case)?

[kbisanz]

Unfortunately, Cal needs GEO-VIIRS-OBC-IP from Geo, so it would have to be run at some point. Though it could be output to disk and then read back in by Cal.

I have emailed you a patch file. If it works (it should), we'll have UW post it to the download site.

[houchin]

Unfortunately, Cal needs GEO-VIIRS-OBC-IP from Geo, so it would have to be run at some point. Though it could be output to disk and then read back in by Cal.

I believe what I did was change the output type of two things in the geo algorithm to DMS, this being one of them (I can't remember the other). I went through the individual algorithm input/output xml files and checked which things were required by each algorithm.

I got the patch and will test it sometime today. But per your other comment in the e-mail that the verified RDR is optional and that Geo and Cal will run it automatically if needed, I will likely just modify my configuration to do that and then test that;