hidden intermediate files in the algorithm running process?

[jhuangadl]

Virutal appliance is eating up my disk space.

I am now having the ADL_4.0-ScienAppliance-1.7 disk file as large as 361 G already. And it keeps growing over time. I can understand if I install something, it should grow, but if I uninstall anything, its size won't decrease. Is it due to automatic caching? If so, how I can disable such auto caching? Will it influence ADl running?

Right now, I am running ADL for multiple granules. I only save the h5 output files from the aerosol algorithm, and remove all the intermediate files. Unfortunately I do have to run SDR and VCM anc granulation modules to prepare for ancillary inputs. I removed all the possible files that I can obviously understand where they are located in the process, such as the unpacked files etc. However, there might be possibilities that ADL is saving some hidden files over time when I run granules one after another. Any hints whether SDR/VCM/Aerosol codes will generate some hidden files that are not easily seen and remove?

Thanks!

[jhuangadl]

I tried to remove some large files in the Linux guest system,

however,

the ADL_4.0-ScienAppliance-1.7 virtual file did not decrease, but increase instead.

What could be the reason behind?

Does this mean it is the virtual file, that is trying to preserve all my actions, is eating up my disk?

Any solutions please?

[kbisanz]

ADL doesn't have any hidden files, at least not that are coming to mind.

If things segmentation fault frequently, you may be leaving "core" files around. They'd probably be located in the directory in which the executable was invoked. However, I like to think that ADL doesn't core much...at least not frequently.

When you delete things from ADL (inside of the virtual appliance), my gut feeling is that the virtual appliance may not get smaller. I would expect the newly deleted space to be marked as "free". It would then be used the next time disk space was needed inside of the virtual appliance. I think that's how VirtualBox works. I'm not sure how VMware works, which is probably what you're using to "play/execute" the virtual appliance. This prevents the virtual appliance from constantly increasing/decreasing its size everytime a file is created/deleted.

Inside of the virtual machine, I would recommend using the "du -m" command to dump the size of each directory in megabytes. You should at least be able to locate the large directory. A "man du" should give you any info you need.

[jhuangadl]

Thanks, Kevin,
What are exactly the names of those dumped 'core' files? I did have some segmentation faults lately, but I couldn't seem to find any dumpled core files.

I was advised to compress the hard disk. It seems to help a bit, although not releasing lots of space for me.

[kbisanz]

The name of the core file can vary between operating system, but typically starts with "core". It'll probably be located in the directory from where the executable was invoked. A "find . -name core\*" should find any core files in the current directory or any sub directories.

For example, consider this program:

Code: Select all

~ > cat test/causeCoreDump.cpp
#include <iostream>

using namespace std;

int main()
{
   int array[100];
   int x = 10000;
   array[x] = 42;    // Over index the array.

   return 0;
}
~ > 

Running it produces a segmentation fault, but no core file.

Code: Select all

~ > g++44 -g -m64 test/causeCoreDump.cpp && ./a.out 
Segmentation fault
~ > ls core*
ls: core*: No such file or directory
~ > 

The reason no core file is produced is that the setting is currently configured off. If you use the bash shell this is controlled by "ulimit". If you use csh, I believe the command is "limit".

Code: Select all

~ > ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 409600
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 409600
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

You can see the core file size is set to 0. That can be changed to unlimited.

Code: Select all

~ > ulimit -c unlimited
~ > ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 409600
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 409600
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
~ > 

Running the program now produces a core file.

Code: Select all

~ > g++44 -g -m64 test/causeCoreDump.cpp && ./a.out 
Segmentation fault (core dumped)
~ > ls core*
core.15033
~ > 

I believe the number is the process ID of that program. The nice thing about core files is you can load them up in a debugger and see where the problem occurred. The core file contains information about the program, including what line the program was executing and the value of variables.

Code: Select all

~ > gdb ./a.out core.15033 
GNU gdb (GDB) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /npd/kbisanz/a.out...done.
[New Thread 15033]
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `./a.out'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000400673 in main () at test/causeCoreDump.cpp:9
9          array[x] = 42;    // Over index the array.
(gdb) where
#0  0x0000000000400673 in main () at test/causeCoreDump.cpp:9
(gdb) list
4
5       int main()
6       {
7          int array[100];
8          int x = 10000;
9          array[x] = 42;    // Over index the array.
10
11         return 0;
12      }
(gdb) print x
$1 = 10000
(gdb) 

You may or may not be creating core files depending on how your shell is configured.

Did the "du" command reveal any large directories? If not, it may be something involving the virtual appliance rather than ADL itself.

[jhuangadl]

Thank you very much, Kevin, Really appreciate your timely and detailed help! Very very useful.
Now I found those core files. They are indeed large files.
I will try the debugger sometime later. Thanks and enjoy a great weekend!
-- Jingfeng

[kbisanz]

If you don't mind me asking, do you know what application was creating the core files? Was the problem caused by something you were doing wrong or is it something we need to take a look at in ADL? If you're not sure and still have any of the core files around, the "file" command should tell you what application caused the core.

Code: Select all

~/test > g++44 -g -m64 causeCoreDump.cpp -o causeCoreDump.exe
~/test > ./causeCoreDump.exe 
Segmentation fault (core dumped)
~/test > file core.28566 
core.28566: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'causeCoreDump.e'
~/test > 

Maybe there is some name length limit as I see "causeCoreDump.exe" appears to have been shortened to "causeCoreDump.e"

Also, if you don't want the core files to ever be produced, you should be able to disable production with "ulimit -c 0" (for a 0 sized core file).

[jhuangadl]

Thanks, Kevin,
I think it is my harddisk shortage caused the segmentation faults, which is why I had asked about the ever-increasing virtual disk file.
I have moved the virtual machine to a big removable disk, and everything seems to run pretty well.
I have disabled the core generation as you recommended. Very very useful tips.
I talked with Bryan at the TIM meeting yesterday, and we sincerely look forward to the new ADL4.1 that features ancillary data granulation modules in the aerosol controller. Although I am able to manage doing ancillary data granulation through subroutines in the SDR and VCM controllers and I developed some wrappers to run multiple-granules on the virtual appliance, I am very interested to see how the ADL4.1 is improved from ADL4.0. As I learned from the TIM meeting, ADL4.1 sounds very promising. Thanks a lot for all the great work!