Page 1 of 1

how to input directories into ADL5.1

Posted: Tue Oct 13, 2015 3:24 pm
by wzchen
I try to run the ADL5.1. Do I have to use database in ADL5.1? or do I have an option to use the old fashion way?

I used to keep all input directories in the "INFTK_DM_ROOT" variable and simply separated them by ":". However, in the new ADL, it seems that I can't do this anymore. Is there any simple way to input all input directories into database?
[16:03 weizhong@rhw3022 script]$ $ADL_HOME/script/DMS/createSQLiteDMS.csh
This script creates the SQLite DMS for ADL...
INFTK_DM_ROOT=/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-TLE/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-PolarWander/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NCEP/2012-01-20:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NCEP/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NAAPS/2012-01-20:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NAAPS/2012-01-21:/data/data020/weizhong/CSPP_CM/ADL51_BLK2_snap/home/pub/ClearCase/STAR/JPSS/ADL_BLK2/LUTs:/data/data020/pub/NPP_DATA/LUTs/AllLuts_BLK2_ADL5.0_Beta/ANC:/data/data020/pub/NPP_DATA/TILES/NovGroundCCR692Tiles/with-meta-all::::/data/data048/pub/weizhong/temp/run_test_viirs/data/input/withMetadata/ProControllerInputs_VIIRS-SCIENCE-RDR/2012-01-21/d20120121_t0722063_e0723316_b00000/run_set:/data/data048/pub/weizhong/temp/run_test_viirs/data/output/withMetadata/ProSdrViirsCalControllerOutputs/20120121-MDR28_J1_VSDR_test/d20120121_t0722063_e0723316_b00000

/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-TLE/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-PolarWander/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NCEP/2012-01-20:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NCEP/2012-01-21:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NAAPS/2012-01-20:/data/data020/weizhong/run_ADL/data/input/withMetadata/IngMsdOutputs-NAAPS/2012-01-21:/data/data020/weizhong/CSPP_CM/ADL51_BLK2_snap/home/pub/ClearCase/STAR/JPSS/ADL_BLK2/LUTs:/data/data020/pub/NPP_DATA/LUTs/AllLuts_BLK2_ADL5.0_Beta/ANC:/data/data020/pub/NPP_DATA/TILES/NovGroundCCR692Tiles/with-meta-all::::/data/data048/pub/weizhong/temp/run_test_viirs/data/input/withMetadata/ProControllerInputs_VIIRS-SCIENCE-RDR/2012-01-21/d20120121_t0722063_e0723316_b00000/run_set:/data/data048/pub/weizhong/temp/run_test_viirs/data/output/withMetadata/ProSdrViirsCalControllerOutputs/20120121-MDR28_J1_VSDR_test/d20120121_t0722063_e0723316_b00000: No such file or directory.
Thanks,

Re: how to input directories into ADL5.1

Posted: Tue Oct 13, 2015 3:55 pm
by kbisanz
You have to use the database in ADL 5.1 and newer.

To use the new way, set INFTK_DM_ROOT to a directory where you want to store the binary data. Then run $ADL_HOME/script/DMS/createSQLiteDMS.csh like you did. That script will create a new SQLite database at $DB_NAME (to store the metadata) and will create a directory structure under $INFTK_DM_ROOT to store the binary data. It might take a couple of minutes to run.

The easiest way to transfer data from a pre-5.1 DMS (a set of directories) to a 5.1 DMS (database + storage area) is to use $ADL_HOME/script/importExistingFlatFiles.csh. You need to provide that script the list of directories from which data should be in imported. The directories should be separated by spaces on the command line, not colons. Under the hood, that script will combine the .asc and binary files into saveset tar files and then import them into DMS using ImportIntoDms.plx.

When you run an algorithm, the binary output will be saved to DMS at the location printed in the status messages at the end of algorithm execution (if you're running it directly; the chain runners mask that output). Any HDF outputs will be written to $HDF_OUTPUT_PATH/AlgName/.

Since the binary outputs aren't stored in a single directory in ADL 5.1, the best way to determine what has been produced "lately" is to perform a query based on time. For example, you can find items with a CreateDateTime (CDT) greater than (GT) a given time by using the -gtcdt flag to
RetrieveFromDms.plx

Code: Select all

$ADL_HOME/script/DMS/RetrieveFromDms.plx -gtcdt "2015-10-13 20:47:45.000000Z"
the current time can be printed with:

Code: Select all

date -u +"%F %H:%M:%S.000000Z"
Note, in ADL 5.3 (not released yet) importExistingFlatFiles.csh was removed. This is because in ADL 5.3 the DMS import gained the ability to import from .asc and binary files directly (not just from .tar files).

Re: how to input directories into ADL5.1

Posted: Tue Oct 13, 2015 4:13 pm
by wzchen
Is there any similar script to import multiple directories after ADL5.3?
Thanks,

Re: how to input directories into ADL5.1

Posted: Tue Oct 13, 2015 5:29 pm
by kbisanz
In ADL 5.3 you can give ImportIntoDms.plx a directory which contains .tar or .asc/bin files and it will import into DMS. In ADL 5.1 and 5.2, it could only import .tar files. The reason why importExistingFlatFiles.csh was created in ADL 5.1 was to package the .asc/bin files into tar files so they can be imported by ImportIntoDms.plx. Since ImportIntoDms.plx can import .asc/bin files directly in ADL 5.3, importExistingFlatFiles.csh is no longer needed.

To import from multiple directories in ADL 5.3, you can use a script or shell loop and call ImportIntoDms.plx. In the bash shell it would be something like this:

Code: Select all

~ > for dir in dir1 dir2 dir3
> do                         
> $ADL_HOME/script/DMS/ImportIntoDms.plx -path $dir
> done 
In csh and tcsh shells it would be:

Code: Select all

~ > foreach dir ( dir1 dir2 dir3)                                    
foreach? $ADL_HOME/script/DMS/ImportIntoDms.plx -path $dir                      
foreach? end
importExistingFlatFiles.csh was deleted in ADL 5.3 because it could be replaced by a simple shell loop in an effort to reduce unneeded scripts and reduce ADL testing effort when we ship out a delivery.

Re: how to input directories into ADL5.1

Posted: Wed Oct 14, 2015 4:43 pm
by wzchen
I got the following error when I ran createSQLiteDMS.csh
/data/data020/weizhong/CSPP_CM/ADL51_BLK2_snap/home/pub/ClearCase/STAR/JPSS/ADL_BLK2/ADL/tools/bin/DmHmiInitialize.exe: error while loading shared libraries: libDmCoreDb.so: cannot open shared object file: No such file or directory
Do you have any idea where this file is?
Thanks,

Re: how to input directories into ADL5.1

Posted: Wed Oct 14, 2015 5:05 pm
by kbisanz
That library should be in $ADL_HOME/lib and should have been created when doing a "make library" or running a build script.

If the library does not exist try this:

Code: Select all

cd $ADL_HOME/CMN/Utilities/DMS/Core/Db/src
make library
If you have problems building in $ADL_HOME/CMN/Utilities/DMS/Core/Db/src my guess would be your SQLite COTS is not set up correctly. Paul Meade noticed a possible minor issue with that in ADL 5.1, so let me know if you have issues.

If the library does exist, check your $LD_LIBRARY_PATH and make sure you've source'd the envSetup file.

Re: how to input directories into ADL5.1

Posted: Wed Oct 21, 2015 3:35 pm
by wzchen
Could you please provide me a list of all available "N_Dataset_Type_Tag" ? Thanks,

Re: how to input directories into ADL5.1

Posted: Tue Oct 27, 2015 3:51 pm
by kbisanz
The available N_Dataset_Type_Tag values are: ANC, ANC-Gran, ANC-Int, AUX, AUX-Int, AUX-Ext, DQN, EDR, GEO, GridIP-Gran, GridIP-Gran-Int, GridIP-Tile, GridIP-Tile-Int, IP, RDR, SDR, TDR, or Verified-RDR. Most of them are fairly obvious. LUTs, processing coefficients, and DQTTs are all type AUX.

I updated the usage of $ADL_HOME/script/DMS/InsertIntoDms.plx to reflect the available types in the ADL 5.3 version that will be delivered.