HSI Special Topics - Tricks and Tips

This page describes some of the ways in which HSI has been used to perform non-obvious functions. If you have a request for a topic that is not covered in this section, or you have a nifty trick to share, please send email to the HSI team so that we can post it here.


  • It is often desirable to retrieve a file from HPSS immediately after storing it, in order to compare the HPSS file with the original local file, or to view the contents of an HPSS file using a tool such as "less".  The most obvious way to do this is to retrieve the HPSS file to a local file, and then use the standard operating system utilities to operate on the local file.


    However, HSI provides the ability to pipe the output of a "get" command into a pipeline of one or more operating system utilities without having to write the file back to the local filesystem.  This can help to avoid problems with disk quotas, insufficient local filesystem space, non-writable filesystems, etc. and can also be significantly faster, depending on the filesystem type, than copying the file to a local filesystem and then operating on it.

     

    Pipe syntax is described on the "Using PIPES for Input/Output" page.


    Here's an example of using a pipe to run the "cmp" utility to compare a local file with an HPSS file:

     

             put my_local_file : my_HPSS_file

             get "|cmp - my_local_file" : my_HPSS_file

     

    In this example, "|cmp - my_local_file" is the local file to which the HPSS file called "my_HPSS_file" is to be renamed, using the local : HPSS rename syntax.  Since the local filename contains spaces, it must be enclosed in quotes.  In addition, since the local filename starts with "|", HSI interprets the remainder of the local filename as a pipe command, and forks a shell to receive the output of the command.   

     

    In this case, the "cmp" command interprets the first argument, "-", as standard input.  Other utilities that read from standard input may use a different parameter to represent standard input, or may simply read from standard input if no filename arguments are provided.

     

    The "-q" command line option, or the "progress" command toggle, can be used to disable file transfer progress messages that can interfere with the output of utilities such as "head".

     

    Note:  It is necessary to type "exit" when piping output into some utilities, such as head, in order to exit from the utility back to the HSI command prompt, even if the entire HPSS file is piped into the utility in less than one full screen.



  • By default, the "get" command will retrieve the specified list of files in tape and position order.  Also note that it may be necessary to use the "-A" option if auto-scheduling has been disabled at the site.

    However, this only applies to a single "get" command - if multiple "get" commands are issued, each command is independent of any others, and only the list of files specified by the single "get" command will be optimized for retrieval. .

    The easiest way to optimize a single "get" command is to use the "here-document" form of the command, by creating an IN file that looks like this:

    get << EOF
       file1
       file2
       file3
    ...
    EOF


  • A common problem is working with filenames containing spaces and special characters that are normally used for pattern-matching. An example of this is shown below:

    Run-202 Future Trends in Automobile Design.pdf

    Pathnames that contain whitespace characters can be enclosed in single or double quotes, or the whitespace characters can be escaped by backslashes.
    For example:

    ls "Run-202 Future Trends in Automobile Design.pdf"
    or
    ls Run-202\ Future\ Trends\ in\ Automobile\ Design.pdf

    A more difficult situation is a filename that contains characters that are normally used for filename globbing.  For example:

    Robotics[102]: Design and Development

    This pathname contains both spaces and square brackets, which are normally used as pattern-matching characters (see "Wildcard characters").  To avoid pattern-matching by the shell (ksh, csh, ...), filenames can be enclosed in single quotes. However, using single quotes for this pathname does not work as it would with the shell , because HSI's command parser does not currently currently distinguish between single and double quotes. (note: this will be corrected in a future version).

    The simplest way to deal with filenames that contain wildcard characters is to disable filename globbing by issuing the "glob" command.  This command is a toggle - each time it is issued, its state switches between "enabled" and "disabled". The command may be issued any number of times within a single HSI session.  The initial state when HSI starts up is "enabled". When filename globbing is disabled, HSI does not check for wildcard characters when it is matching filename patterns, instead, filenames are matched literally, character-for-character.  

    For the example above, the following could be used:
    1. disable filename globbing
       glob  
    2. issue command(s) (Note that quotes are still needed because of whitespace characters in the HPSS pathname)
       ls -l "Robotics[102]: Design and Development"
       get roboticsCourse : "Robotics[102]: Design and Development"
    3. re-enable filename globbing
       glob