happened again. run ADL4.1+Mx7.1 problems

[wzchen]

I installed Mx7.1 without any problems. I got all 48 executable files under bin directory. I test ATMS-SDR. It runs OK. However, it failed on VIIRS SDR. I couldn't find any clue from the log file.
I have no idea where to go next step.
Thanks,

[kbisanz]

The log file appears to just stop in the middle. Did the process stop abnormally, such as a segmentation fault?

Are you running via the command line or the ADL chain runner (command line or GUI versions)?

It appears you're running an operational granule: NPP000550715754. Verify that $NPP_GRANULE_ID_BASETIME is set to 1698019234000000 for operational data. You can change that in $ADL_HOME/build/envSetup.ksh

[wzchen]

Yes, it was stopped abnormally. I am using ADL chain runner in command line mode. The $NPP_GRANULE_ID_BASETIME has been set to 1698019234000000 in envSetup.ksh file. Since it can ran ATMS, I supposed that the ADL has been installed correctly. The attachment is the screen output from the run. I remembered that I got this kind of problem because of the missing SDR LUTs. However, LUT directory has been token correctly in the "INFTK_DM_ROOT" variable.

[kbisanz]

Have you seen any core files laying around? They are probably in the directory that the chain runner was started from, but I'm not sure. If you have a core file, can you print a stack trace. You can do this with gdb by doing a "gdb /path/to/executable /path/to/corefile" and then typing "where" at the gdb prompt.

Also, when running, can you turn the debug level to LOW and post the log file?

Does this happen on all VIIRS granules or just 1?

[wzchen]

ADL is working fine now. The previous problems disappeared. Also, the log file looks fine. Now, I can grep error messages from it. I am not sure why and how it caused the problems but it's gone. I didn't change anything. Some granules which had problem previously went through OK. It's weird.
Thanks.

[wzchen]

The same problem happened again today. However, I was able to run the same granule with the Mx6.7 built on the same machine in the same terminal. The new Mx7.1 built just can't go through. All granules were failed. I am using command line TK chain runner. Here is the log file with LOW level on.
BTW, I can't find the core files. Do you have idea where it is saved?
Thanks,

[wzchen]

I can not upload attachment. Could anyone fix the problem please?
Thanks.

[kbisanz]

Regarding the attachment issue, there is probably a size limit on the attachment. Try using gzip to compress it. That *might* compress it enough. If that does not work, you can upload the log file to ftp://ftp.ssec.wisc.edu/pub/incoming/ Note that files uploaded there are automatically removed after a few days. Uploading the file is probably a better option that attaching a huge file.

Regarding the location of the core files, make sure the production of core files is turned on. You can check this with "ulimit -c" in the bash shell.

Code: Select all

~ > ulimit -a | grep core
core file size          (blocks, -c) unlimited
~ > 

The "unlimited" is the maximum size of the core file. A size of 0 would disable core file production. You can set the core file size with something like "ulimit -c 100" or "ulimit -c unlimited". Note that 100 is not large enough, just an example. Also note that you may not have permission to increase the core file size. You would need to talk to your system administrator. If the size is 0, that is why you are not seeing core files. If the size is not zero, they are being placed someplace.

Once you know that core file production is enabled, I would check the following locations:
--The TEMP_LOC property as specified in $ADL_HOME/.lw_properties
--The directory where you started the chain runner from
--$ADL_HOME/bin

A random question: Are you using any of the data delivered with the Mx7.1 delivery to process your operational data? Another user was mixing and matching data operations and the Mx7.1 delivery and it was causing issues.

Please post back with your results.

[wzchen]

You're right. The original log file was too big. Now, I can attach a failed log file from last week.

Yes, my core file was turned off. I will set it to unlimited from my .bashrc file.

The command line chain runner still didn't run this morning. However, it worked again after I tried the following steps:
I re-ran the sdr algorithm executable file directly using the xml file which was generated by the failed command line chain runner. It worked!
Then I set the same "INFTK_DM_ROOT" variables in the chain runner's GUI mode. It worked too.
After that, I went back the command line chain runner. It worked now. And I don't know why. Could you please test command line chain runner? My scripts rely on it.

I will post back when it happens next time. I feel it will happen soon.

Thanks,

[wzchen]

OK. It happened faster than I thought. Here is the new log file.

I set core size to 'unlimited'. However, I can't still find it in all those 3 locations you listed.

Thanks,