CSPP-GEO key APID errors

ve2fxl
Posts: 14
Joined: Sun Nov 29, 2020 2:59 pm

CSPP-GEO key APID errors

Post by ve2fxl »

Hi,

My CSPP-Geo/Geo2grid is working 24/24h since a quite long time and I wasn't checking much until the Holidays. I noticed that in my GRB.log files, I have quite often a lot of the following errors:
jvm 1 | 2023/01/27 16:59:01.027 | Should never happen, invalid product key:
jvm 1 | 2023/01/27 16:59:01.027 | [ERROR] (GRBReceiver) Unable to route packet for key APID: a1, last product time seen: 0x0a1_2023-01-27T21:56:17.341

And APID a1 is CONUS Channel 02 and here's result I always get, large missing block in the bottom right. (File attached for the corresponding image of the error in GRB.log)
I see also in GOES-16 Image Viewer of Star Nesdis that they get the same missing block like me but only on the True Color composite, not channel 02 which is weird as True Color is made from C01, C02 and C03. It's doing that for a lot of CONUS per day and it began at least in the december or before.

And, I also get about 4 to 6 completely missing C02 Mesos for Meso-1 and also Meso-2 so about 8 to 12 per day for both. But I can't see them as missing in Star Nesdis. Also, they don't miss when using GRBImager, a software from Brett Casebolt (N6BY). It began to happen regularly starting on November 28th 2022. So could it be that something changed on that date in the GRB stream that CSPP-Geo is not aware? The corresponding errors are:
For Meso-1 missing C02 channel:
jvm 1 | 2023/01/27 19:48:35.528 | Should never happen, invalid product key:
jvm 1 | 2023/01/27 19:48:35.528 | [ERROR] (GRBReceiver) Unable to route packet for key APID: c1, last product time seen: 0x0c1_2023-01-28T00:48:25.208

And for Meso-2 missing C02 channel:
jvm 1 | 2023/01/27 08:09:05.663 | Should never happen, invalid product key:
jvm 1 | 2023/01/27 08:09:05.663 | [ERROR] (GRBReceiver) Unable to route packet for key APID: e1, last product time seen: 0x0e1_2023-01-27T13:08:55.180

There's no bad CRC when that happens, it's not bad signal reception. And why always Channel 2? If it was random, I would missing a lot in Full Disk and CONUS due to the huge amount of data per hour for those sectors and why the whole image, not just little blocks?

Do you get both CONUS Channel 02 missing blocks and whole Meso-1 and Meso-2 sometimes missing channel 02 with cspp-geo on G-16? I know also someone that has the same large missing blocks in the bottom right in Conus Channel 02 on GOES-18 with CSPP-Geo. Please help. I would also like to see a 1 or 2 days log of your GRB.log at University of Wisconsin! Thanks!

Regards,
Luc
VE2FXL
Attachments
GOES-16_ABI_RadC_C02_20230127_215617_GOES-East.jpg
GOES-16_ABI_RadC_C02_20230127_215617_GOES-East.jpg (3.99 MiB) Viewed 13835 times
jbraun
Posts: 102
Joined: Thu Sep 18, 2014 11:25 am

Re: CSPP-GEO key APID errors

Post by jbraun »

Hi Luc,

Sorry to hear you are having problems. Can you first tell us what version you are running of the CSPP Geo GRB software?

Provided your $PATH is set (export PATH=$PATH:$HOME/cspp-geo-grb-1.0 )

you can run the following commands to verify the version you are running:

cd $HOME
grep VERSION ./cspp-geo-grb-1.0/GRB-R/grbr/config.py

Thanks,
Jess
ve2fxl
Posts: 14
Joined: Sun Nov 29, 2020 2:59 pm

Re: CSPP-GEO key APID errors

Post by ve2fxl »

Hi Jess,

My version is latest: 1.0.27

I can see that missing blocks or channels are missing also on Star Nesdis GOES Viewer so I suspect that they are using CSPP-Geo to ingest or there's an issue with the uplink stream that CSPP-Geo can`t handle but GRBImager can! Can you see those errors on your side with CSPP-Geo?

Regards,
Luc
jbraun
Posts: 102
Joined: Thu Sep 18, 2014 11:25 am

Re: CSPP-GEO key APID errors

Post by jbraun »

Hi Luc,

Thanks for verifying your version. While were were waiting to hear back from you we started to check out our internal systems and we were able to come across the same missing block in the image you provided in your first post on one of our processing systems running 1.0.27, along with similar errors that you included. I've noted this information to the rest of the CSPP Geo GRB team and we are investigating further. We will get back to you once we have some additional information.

~Jess
ve2fxl
Posts: 14
Joined: Sun Nov 29, 2020 2:59 pm

Re: CSPP-GEO key APID errors

Post by ve2fxl »

Hi Jess,

Seems like the errors went away 2 days ago. I asked espcoperations about that before I asked here and they didnt really answer but maybe that's related! I asked them and I hope it stays that way

Thanks
Luc
tommyj
Posts: 7
Joined: Tue May 22, 2018 8:33 pm

Re: CSPP-GEO key APID errors

Post by tommyj »

Hello Luc - First, on behalf of the CSPP team I want to thank you for taking the time to contact us on the CSPP forum, and report the issue you were seeing in great detail. This is very much appreciated and helps us in so many ways.

We do believe the problem you reported began in December with the ground system update to DO.11.02.00, and stopped with the reversion to DO.09.07.00 on January 30th, as described here. This is admittedly speculative at the moment, based on statistics we have seen here at SSEC.

You may not care much about the technical details of the problem, but I will give you a brief description here regardless. The GRB data transmission protocol follows well-defined patterns, of course. One of these patterns is that, for each data product (e.g. an ABI CONUS image), all the image data is transmitted first, followed by all the image metadata. We discovered that occasionally, this rule is violated and some product image data will trickle in after the final metadata. We determined that by waiting a very small amount of time (~0.5 seconds), over 99% of the time we could gather and patch in any missing product data. During the period you reported, over the holidays and up to Jan 30th, for those rare cases when an image was not transmitted correctly, the time needed to reclaim all late product data crept closer to 2.0 seconds. In those cases the CSPP Geo GRB software would lose that data, resulting in the "unroutable packets", "missing corners" etc., which you saw and reported.

We hope of course this scenario will not reoccur, but are discussing internally and taking measures to deal with it in our next release if it should reoccur. Thanks again Luc and let us know if you have further questions or comments on this or any other topic.
ve2fxl
Posts: 14
Joined: Sun Nov 29, 2020 2:59 pm

Re: CSPP-GEO key APID errors

Post by ve2fxl »

Hi Tommy,

I'm very glad to help and I wonder why nobody found that before me especially weather organizations. And I'm very grateful that you found what was happening. As I read the NOAA satellite message you referred to, I suspect the errors will be back starting on March 1st as they will probably reinstate the software patch so yes it would be great that CSPP-GEO would be patch to receive without errors when that happens again. And as I says, another software is able to receive the stream without any error over 24 hours so there's something to do with them.

As they reverted back, I began to receive randomly some CRC check failed errors and those usually makes appear a thin and short black line on FD image, I haven't checked on which channel yet by the way and it's not on my streamer that it's happening and if that happens, my streamer reports it but for the hour I send below, there wasn't:
jvm 1 | 2023/02/02 07:09:35.301 | [ERROR] (CrcDecoder) CRC check failed on data (non-Idle) frame
jvm 1 | 2023/02/02 07:09:35.301 | [ERROR] (CrcDecoder) CRC check failed on data (non-Idle) frame
jvm 1 | 2023/02/02 07:09:35.301 | [ERROR] (CrcDecoder) CRC check failed on data (non-Idle) frame
jvm 1 | 2023/02/02 07:09:35.301 | [ERROR] (CrcDecoder) CRC check failed on data (non-Idle) frame
jvm 1 | 2023/02/02 07:09:35.301 | [ERROR] (CaduService) Frame Error, VCID: 5, prev VCFC: 848333, this VCFC: 848338
jvm 1 | 2023/02/02 07:09:35.301 | [ERROR] (GRBReceiver) APID: 0xda sequence error, counter: 14710
jvm 1 | 2023/02/02 07:09:35.301 | [ERROR] (GRBReceiver) APID: 0xd0 sequence error, counter: 14031

Thanks and please keep me informed of developments!

Kindest regards,
Luc
VE2FXL
PS: Receiving GRB at over 46 Latitude North with 2.1m solid dish and average signal over 11dB SNR in Quebec.
jbraun
Posts: 102
Joined: Thu Sep 18, 2014 11:25 am

Re: CSPP-GEO key APID errors

Post by jbraun »

Hi Luc,

The default bundle tie-off delay was reduced to 0.5 seconds in Version 1.0.24, consistent with the recommended delay in the GOES-R Product Users Guide, Vol. 4, Section 5.0.

However, it currently is possible to change this default if you so choose. Section 8.8 of the CSPP Geo GRB Software Users Guide will explain how to change this value.

We will be reporting this issue back to the GOES-R program and if any additional changes are required in the future as a result, we will provide updates as needed through the CSPP Geo website.

Thanks,
~Jess
ve2fxl
Posts: 14
Joined: Sun Nov 29, 2020 2:59 pm

Re: CSPP-GEO key APID errors

Post by ve2fxl »

Hi Jess,

Thanks for the added informations. So if the errors are back again, I'll try to change the setting you referred me. And by the way, which new value do you recommand? 2 or 10 seconds?

Regards,

Luc
Last edited by ve2fxl on Sun Feb 05, 2023 12:31 pm, edited 1 time in total.
jbraun
Posts: 102
Joined: Thu Sep 18, 2014 11:25 am

Re: CSPP-GEO key APID errors

Post by jbraun »

Hi Luc,

We would recommend going back to the old default that CSPP Geo GRB used to have - 10 seconds. We are in communications with the GOES-R program about the previous errors you reported and from our analysis, the latency can vary above 2 seconds. Our team will continue discussing this issue with the program and will also investigate any future changes to our software to help mitigate the issue better if it occurs again.

~Jess
Post Reply