Workbook for SLD Offline Users - UNIX Staging

The UNIX tape staging system provides a convenient means to store large files on tape and to retrieve those files to disk for temporary use. It is particularly useful in dealing with large data sets such as PAW ntuples. The steps involved for storing your ntuples are:

Some less important related topics are:

Store Your Ntuple in Temporarily Disk Space

The /nfs/ravel/u1/ partition is available for temporary storage of files. Create a directory (using mkdir. Use the same name as your account name) and store your files there. They will automatically be erased after 3 days.

You need to be in the EY group to have write access to the disk. Ask Richard Dubois if you find you don't have write access.

Get an SLD tape

You can get a tape from the SLD pool using the GETFREE utility from VMS or VM.

Copy the Disk File to Tape

sldtpwrite -vTAPENAME -q1-NFILE -lsl -FF -L32768 -gCART FN1 FN2 ...

TAPENAME = Silo tape name, eg qa0072
NFILE = # files to copy, eg 4
CART = a fixed keyword, not to be substituted
FN1 = filename of first disk file
FN2 = filename of second disk file
Filenames need to be fully qualified, eg /nfs/ravel/u1/richard/MyNtuple.rzdat.

sldtpwrite is available from the SLD group area /afs/slac/g/sld/bin. You will need to have set up the standard SLD .sldcshrc environment to use sldtpwrite. It checks the SLD Oracle database to check ownership of the tape.

This command takes a bit of time, so it might be a good idea to run it in batch or in background. Be sure to keep an error file, and check that it runs OK.

Copy the Tape File Back to Disk

Once you have moved your data from its temporary home on disk to tape, you'll need to move the data back to a disk file before using it. The command to do this is:

stagein -VTAPENAME -qFILENUM -FF -L32768 -sMMM FN

TAPENAME = Silo tapename, eg qq0072
FILENUM = file number on tape, eg 1
MMM = maximum disk file size in MB, eg 300
FN = filename which will be used for this data

Data will be copied from tape to the staging disk, which is dedicated to this useage. For normal operation, you do not need to know the actual name of this disk. There must be sufficient room on the staging disk for a file up to MMM MB in size; otherwise, stagein will fail. Hence, do not specify MMM much larger than it needs to be. I am not sure what happens if the actual file is longer than MMM MB.

The filename FN may point to any disk or partition, even one with practically no free space (eg your home directory), because stagein does not actually put the data there. It will simply contruct a symbolic link from FN to wherever the file really is on the staging disk. When stagein returns, you can direct the analysis program to use FN as input. Note that you can only access the staged files from one of the interactive farm machines (currently the Morgan0n machines).

This command may take some time to execute. If the file is already on disk (eg from an earlier stagein command by you or somebody else), stagein should know about it and immediately return with FN set up to link to the file.

The disk file will be purged by the staging system when it needs room for other datasets; however, the link FN is your responsibility. Use the rm command to get rid of it. Note that the real disk file could be gone even if you have a link pointing to it, i.e. always execute the stagein command, and do not rely on the presence of a link.

Getting Rid of a Staged Disk File

You can only stageclr your own files. Please do not use the standard Unix command rm for this purpose. The space freed up may not be known to the staging system.

Use stageqry -s to see how much space is available in the staging system.

If you want to see which tapes are active, issue the command tpstat. Unlike the other commands, you must be logged onto the Unix machine known as silosrv1.

Reserving Staging Space for Writing

This is useful when copying data from 8-mm tape to disk. It can, of course, be used for any output file.

stageout -VTAPENAME -qFILENUM -lnl -FF -L32768 -sMMM -poutput FN ...

TAPENAME = (arbitrary) tapename, e.g. FOO
FILENUM = (arbitrary) file number, eg 1
MMM = space to be reserved in MB, eg 300
FN = filename associated with it, e.g. BAR

When this command returns successfully, it will have created a link from FN to an actual disk location. You can then simply write to FN.

Note that you have reserved MMM MB of space, but your actual useage may be more or less. If you try to write more than the reserved space and if there is indeed more space available, it will work.

Back to Workbook: Unix

This page was adapted from Charlie Young's E143 documentation.
Richard Dubois
22 May 1995