Using PIPEs for Input/Output


HSI supports the use of Un*x "pipes" when either storing or retrieving files, using the standard rename syntax:


    get local_File : HPSS_File

or

   put local_File : HPSS_File


Using STDIN and STDOUT for Input Output


For "put" commands, if localFIle is a minus sign "-", then HSI reads input from its STDIN.

This is intended for use in shell script pipelines, such as:


         tar cf - . | hsi options "put - : someTarFile"

(Note: HTAR provides the same capability as the above, but is generally much faster).


However, if used interactively, HSI can also be used to read from the TTY to pipe a file into HPSS.  In this case, a single <ctrl-D> (CTRL key and "d" key pressed simultaneously) can be used to terminate the input (take care not to use more than one <ctrl-D>, as this is also a way to exit HSI).  For example:


O:[hpss-nccs]/home/gleicher/temp->put - : ToDo-Today

Here is my TO-DO list:   (Lines entered from the TTY)

       wake up                

       work

       eat

       go to bed

<ctrl-D>

put  '<stdin>' : '/home/gleicher/ToDo-Today' ( 78 bytes, 0.0 KBS (cos=5081))


For "get" commands, if localFile is a minus sign, then HSI pipes the output to its STDOUT. Again, this is intended for use in shell script pipelines, such as:

         hsi options get - : someTarFile | tar xf -


As explained in HSI Special Topics, Tips and Tricks, this is one way to view the contents of HPSS files without first copying them to a local file, for example, using the above HPSS file:


       get - : ToDo-Today

Here is my TO-DO list:            

       wake up

       work

       eat

       go to bed

get  '<stdout>' : '/home/gleicher/ToDo-Today' (2010/12/20 19:47:38 78 bytes, 221.7 KBS )


Using Un*x PIPE Commands for Input and Output


For both storing and retrieving, if the first character of localFile is a vertical bar "|", HSI interprets the remainder of localFile as a pipe command after skipping any leading white space.  


The pipe command can contain one or more shell commands, such as:

        put "| cat file1.c file2.c " : oneCFile

or

        put "| ls -l | awk '{print $9}' " : listOfFilenames

or

       get "|tar xf - " : tools.tar   

or

       get "|less" : bigDocument  (see note 3 and 4)


Notes:

1.  "localFile"  must be enclosed in quotes if it contains whitespace characters.
2.  To disable HSI file transfer progress messages, which can overwrite parts of the last line of output on the TTY, use either the "PROGRESS" command toggle, or the "-q" (quiet) option on the HSI command line.
3. When using interactive utilities such as "less", it is usually, although not always, necessary to use "exit" (or "end", "quit", or whatever the utility recognizes as a command to terminate) in order to exit the utility, even if HSI is able to pipe the entire file to the utility.  


WARNING:  HSI transfers entire files, using the parallel transfer features of HPSS in 

order to achieve the fastest possible transfer rates.  When reading HPSS files to STDOUT, or to a local pipe command, it is possible to cause a transfer to abort if the

file is longer than two data buffers (8 MB by default), and data is not consumed at a fast enough rate.  This is normally not a problem for utilities such as "cmp" or "md5", but

could potentially be a problem for tty-oriented utilities such as "less", "more", "tail",

etc.