This post describes one way to set up a real-time Level 0 processing system for direct broadcast data from Terra and Aqua. The system described here receives a raw telemetry stream (i.e., "clock and data") from a separate SeaSpace TeraScan reception computer, and uses the RT-STPS application to create Level 0 files in real-time as the data are received from the SeaSpace computer. The Level 0 files are written to disk in the standard PDS format, and each Application Packet ID (APID) type is written to a separate file. When the pass is complete (i.e., signal lock is lost), the Level 0 file is complete and ready for further processing (e.g., by the IMAPP Virtual Appliance). We use a system like the one described here to create Level 0 files from Terra and Aqua data acquired by our own SeaSpace X-band antenna. The processing computer is a dual quad-core Dell system running CentOS 5.3 64-bit.
Cheers,
Liam.
PS: Note that the Simulcast system is *not* required for real-time processing of Terra and Aqua data. We run Simulcast here at UW because we supply real-time data to the Simulcast client from the NASA DRL.
---
HOW TO CONFIGURE RT-STPS AND SIMULCAST FOR REAL TERRA AND AQUA PROCESSING
Liam.Gumley@ssec.wisc.edu
2011/02/28
-------------------------------------------------------------------------------
1. Install Sun Java6 JDK
If you get this, it means Sun Java is not installed or configured yet:
$ java -version
java version "1.4.2"
gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-44)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- CentOS 5.3 64-bit (run this script via sudo or as root)
#!/bin/bash
# Script to install Sun Java6 Development Kit for 64-bit Linux on CentOS 5.3
# Note that Sun Java binary was obtained from http://java.sun.com/javase/downloads/index.jsp
# Liam.Gumley@ssec.wisc.edu 17-APR-2009
cd /opt
sudo wget ftp://ftp.ssec.wisc.edu/pub/gumley/java ... ux-x64.bin
sudo /bin/sh jdk-6u13-linux-x64.bin
sudo /usr/sbin/alternatives --install /usr/bin/java java /opt/jdk1.6.0_13/bin/java 2
sudo /usr/sbin/alternatives --config java
- Ubuntu 8.10 64-bit
$ sudo aptitude install sun-java6-jdk
In both cases (CentOS or Ubuntu, you should then get something like this):
$ java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
-------------------------------------------------------------------------------
2. Install RT-STPS and SIMULCAST
cd $HOME
mkdir data
mkdir tarfiles
cd tarfiles
lftp -c "get -c ftp://drl-fs-1.sci.gsfc.nasa.gov/.SOFTW ... 4.1.tar.gz"
lftp -c "get -c ftp://drl-fs-1.sci.gsfc.nasa.gov/.SOFTW ... bundle.zip"
cd $HOME
tar zxf $HOME/tarfiles/rt-stps4.1.tar.gz
mkdir simulcast
(cd simulcast; unzip $HOME/tarfiles/sc-4.1-bundle.zip)
-------------------------------------------------------------------------------
3. Configure port 4935 to allow incoming telemetry stream (for RT-STPS)
- CentOS 5.3
sudo /sbin/iptables -I INPUT 1 -p tcp -s golfball.ssec.wisc.edu -d sunset --dport 4935 -m state --state NEW -j ACCEPT
sudo /etc/init.d/iptables save
-------------------------------------------------------------------------------
4. Start RT-STPS and load satellite configuration file
$HOME/scripts/rtstps_startup.bash
$HOME/scripts/rtstps_configure_terra.bash
RT-STPS is now ready to receive incoming telemetry (clock and data) from Terra on port 4935. On a SeaSpace TeraScan system, you
can send a pass to the RT-STPS computer like this:
send_pass send_header=no input_pass=3 port_number=4935 client_mode=yes host_name=sunset connect_tries=3 stream_proc=""
The decoded Level-0 data will appear in the $HOME/data directory.
-------------------------------------------------------------------------------
5. Configure the acquisition computer to send live telemetry
In this case, we have configured a SeaSpace TeraScan X-band acquisition computer to send live telemetry to the
host running RT-STPS. Note that TeraScan version 3.2 (April 2002) or higher is required to support the "send_pass"
functionality described below.
On the SeaSpace system, here is how to edit the configuration.
cd /opt/terascan/pass/config
cp system.config system.config.backup (keeps a backup of original system configuration)
Then edit the lines containing the string "recvhost" to look like this:
[recvhost-1]
recvhost.name: sunset.ssec.wisc.edu (enter your system name or IP address)
recvhost.active: yes
recvhost.bandwidth: 50
recvhost.assoc_chains: 1 2 3 4 5
recvhost.send_live: yes
recvhost.stream_proc:
recvhost.port: 4935
recvhost.tries: 3
recvhost.send_header: no
The meaning of these fields can be found in system.config.README, i.e.
# tscan_send send_pass configuration
# tscan_send is started automatically by tscan_pproc if there are any configured instances,
# active or not; configuration order determines run order.
#
# [recvhost-#] tscan_send config number, positive integer
# recvhost.active: yes or no
# recvhost.name: host name or ip address of destination host
# recvhost.bandwidth: bandwidth percentage
# recvhost.assoc_chains: list of associated telemetry chain numbers, blank separated
# recvhost.send_live: yes or no
# recvhost.stream_proc: e.g., rs for CCSDS Reed-Solomon (optional)
# recvhost.port: network port number, default is 1480
# recvhost.tries: number of connect attempts, default is 3
-------------------------------------------------------------------------------
6. Configure the acquisition computer to send Terra/Aqua configuration commands to the RT-STPS computer
RT-STPS cannot tell which satellite telemetry stream is being received. It must be configured explicitly before
each pass commences. This can be done in the following way on a SeaSpace TeraScan acquisition computer.
today=$(date +%Y/%m/%d)
echo 'ssh ipopp@sunset /home/ipopp/rt-stps/bin/load.sh /home/ipopp/rt-stps/config/terra.xml' > command_file
listsched | grep $today | grep sched | grep terra | cut -c51-55 | awk '{print "at -f command_file "$1" - 5 minutes"}' > Script
source Script
echo 'ssh ipopp@sunset /home/ipopp/rt-stps/bin/load.sh /home/ipopp/rt-stps/config/aqua.xml' > command_file
listsched | grep $today | grep sched | grep aqua | cut -c51-55 | awk '{print "at -f command_file "$1" - 5 minutes"}' > Script
source Script
This command can be run in a crontab on the acquisition computer at 00:05 UTC every day.
-------------------------------------------------------------------------------
7. Start Simulcast
First, edit the router-config.xml and server-config.xml files in simulcast/sc/configs to reflect your station name and location.
$HOME/scripts/simulcast_startup.bash
The output files from Simulcast will appear in $HOME/simulcast/sc/spool/server
-------------------------------------------------------------------------------
8. Set RT-STPS and Simulcast to start on system boot
Edit /etc/rc.local to contain the following lines:
# RT-STPS startup (LG 2009/04/20)
su - ipopp -c "/home/ipopp/scripts/rtstps_startup.bash"
# Simulcast startup (LG 2009/04/20)
su - ipopp -c "/home/ipopp/scripts/simulcast_startup.bash"
-------------------------------------------------------------------------------
9. Set up crontab to restart RT-STPS and Simulcast if needed
Every now and then, RT-STPS and Simulcast will quit with no fanfare. On our system, it seems to happen
a few times per year. To make sure these services remain operational a script was created to check
if RT-STPS and Simulcast are running, and restart them if needed. The script is run every hour via the
following crontab entry:
0 * * * * $HOME/scripts/check_rtstps_simulcast.bash
The output from this script goes to $HOME/logs/rtstps_startup.log and $HOME/logs/simulcast_startup.log.
-------------------------------------------------------------------------------
10. Set up disk scrubbing schedule for RT-STPA and Simulcast
The directories used by RT-STPS and Simulcast must be scrubbed routinely to avoid filling the host filesystem.
A script for this purpose is run via crontab every night at midnight via the following crontab entry:
0 0 * * * $HOME/scripts/scrub_data.bash
All files older than 48 hours will be removed from these directories:
dir=$HOME/data/
dir=$HOME/simulcast/sc/spool/server/