Using UUENCODE/UUDECODE in BatchLogin sessions.

[Back] Transporting files may on the surface appear to be one of the most trivial things. Unfortunately, when it comes to automated tasks, the complexity of a non-homogeneous network can become an enormous issue. Simply getting a file from point A to point B becomes hideously troublesome, as differing states of security, one-way gateways and differing file transport mechanisms prevent a mass of servers from being treated as a similar whole.

BatchLogin leverages UUENCODE and UUDECODE for transporting files across the link, meaning that even if most protocols are broken along the way, simply the serial connection between two boxes alone can be utilized. This means that simple scripts can push a file without worrying about the means to get it there. There are two ways described in this document. One way is to have a script echo out a specific text string, which is captured by BatchLogin as a signal to transport a file. The other way involves calling the transport routine directly in an expect script. Both ways work equally well, although the expect function call is the cleanest and most flexible.

Using the Expect Function Calls

The two functions are:

get_remote_uuencode REMOTEFILE LOCALFILE
put_remote_uuencode LOCALFILE REMOTEFILE

Users that call these functions should be aware of how they work and their limitations. The main limitation is one of speed. Anything over a few megabytes is likely to take an unbearable amount of time, and the speed is extremely slow over a telnet link. Also, the method of operation, although similar, is necessarily different: Getting files from a remote system results in the files accumulating in a directory with the name of the label appended. Pushing the files to a remote system results in an exact method of operation, whereby the files are named exactly as called in the reference statement.

A Push File Example

User Eric wishes to roll out a remote agent with an expect script. He copies the "SHELL.exp" in his ~/.batch2/scripts directory to a script called "rollagent.exp". The .exp extension signals BatchLogin that this isn't a shell script to be copied to the remote hosts upon which it must execute -- rather, expect will merge this script with the main BatchLogin function at runtime, inserting the statements between the labels:

### BEGIN_POSTPROCESS_GUTS
# my preprocessing goes here
### END_POSTPROCESS_GUTS
   ### BEGIN_SCRIPT_GUTS
   ##
   # my expect program goes here
   ##
   ### END_SCRIPT_GUTS

### BEGIN_POSTPROCESS_GUTS
# my postprocessing goes here
### END_POSTPROCESS_GUTS

Note that the case, position and BEGIN/END statements are unique and everything outside of them ignored as far as expect and BatchLogin are concerned. This is a special BatchLogin script -- it will not work as an expect script directly without modification.

The preprocessing and post-processing are expect statements that will be executed before looping through a group of servers, and after, respectively. The "SCRIPT_GUTS" portion of the program contains the primary program that is inserted into the loop. An expect call can be executed here, and that includes function calls, of which, BatchLogin has many.

Eric proceeds to insert a statement to initiate a file transfer of the agent, which is being delivered as a .tar.gz file:

### BEGIN_POSTPROCESS_GUTS
# my preprocessing goes here
### END_POSTPROCESS_GUTS
   ### BEGIN_SCRIPT_GUTS
   ##
   # transfer the agent to the remote machine:
   put_remote_uuencode  /export/depot/agents/newagent.tar.gz   /tmp/newagent.tar.gz
   
   ##
   ### END_SCRIPT_GUTS

### BEGIN_POSTPROCESS_GUTS
# my postprocessing goes here
### END_POSTPROCESS_GUTS

But it's not enough to transport the agent, it needs to be unpacked and installed as well:

### BEGIN_POSTPROCESS_GUTS
# my preprocessing goes here
### END_POSTPROCESS_GUTS
   ### BEGIN_SCRIPT_GUTS
   ##
   # transfer the agent to the remote machine:
   put_remote_uuencode  /export/depot/agents/newagent.tar.gz   /tmp/newagent.tar.gz

   # Unpack the archive
   send -- "cd /tmp\r"
   get_prompt 
  
   send -- "gunzip newagent.tar.gz\r"
   get_prompt 

   send -- "tar xvf newagent.tar\r"
   get_prompt

   send -- "cd agentInstall\r"
   get_prompt

   send -- "./installagent --foo --bar --etc --etc\r"

   # look for specific output from the install program
   expect { 
      "Install successful" { 
          send_user_p "Agent installed on node $NODENAME ."
      }
      timeout { 
          send_user_p "Agent instal failure on  $NODENAME ."
          run_interactive
      }
   }

   ##
   ### END_SCRIPT_GUTS

### BEGIN_POSTPROCESS_GUTS
# my postprocessing goes here
### END_POSTPROCESS_GUTS

Note that this is one way to do it (somewhat painfully, as expect(TCL) isn't known for being the most intuitive program on the planet. Eric tests his program on a known host, until it's dialed in and working as expected (pun intended). He's now ready to roll it out across a bunch of machines.

A Push File Example (revisited)

It's also possible to simply write a couple of small shell scripts: One that tells expect to call the function above, and another one that does the installation. Since the shell execution is easier, more intuitive, and in general, more predictable, it's by far the easier method.

Eric creates two scripts, one is called "Apushagent.sh" and the other is called "Binstallagent.sh". The Install script is left up to the imagination. It relies only upon the tar archive being on the box. The Apushagent.sh script is simple:

#!/bin/ksh
echo "BATCHLOGIN_LOCAL_TO_REMOTE_TRANSFER /export/depot/agents/newagent.tar.gz TO /tmp/newagent.tar.gz END_TRANSFER"

Yes, that's it. This script must execute before the install script. At runtime, BatchLogin will call the first script, which echoes the text shown, which expect "sees" as a call to action: In this case, it invokes the routine to upload the tar-ball into place. The next script that is run is the installer, which can contain as much or as little logic as any other script. The beauty of this method is that the script can be exactly the same one tested on another host.

Note: The names here are vitally important. The "A" at the front of the push script and the "B" at the front of the install script ensure that they're executed in alphabetical order at runtime.

The easiest way to accomplish this task is to use the Java GUI (option 3 in the bl2 main menu). Select the passlist to execute against, and the scripts to transport. It's important that they be ordered correctly, and it's done in alphabetical order, so you'll have to make sure that your scripts are named appropriately for the proper execution. After the first script completes, and before the second one executes, the call to put_remote_uuencode will be executed.

A Pull File Example

Pulling is similar to pushing, except that files by their nature must be collected in one location with differing names.

To accomplish this, the expect script places the label at which the file was collected at the end of the file. The label in this case is the one in the passlist or lists upon which the call was executed. Here's a simple example that collects all of the kernels from several Unix machines into one location:

#!/bin/ksh
echo "DATA $1 $(uptime)"
echo "BATCHLOGIN_REMOTE_TO_LOCAL_TRANSFER /vmunix TO /export/data/kernels/vmunix END_TRANSFER"

The files are collected from a passlist that has three servers labeled: prod1: qv1: and test1:. After execution, the directory /export/data/kernels/vmunix on the BatchLogin execution host contain the following three files: vmunix.prod1, vmunix.qv1 and vmunix.test1. This is the way that BatchLogin accounts for the same file from multiple hosts.

Operation: How Does This Work, Exactly?

Basically, uuencode and uudecode are extremely old commands, implemented as part of uucp (Unix to Unix copy). The commands are extremely old, and extremely basic in operation. They take a binary file, and reduce it to a stream of known text characters, 60 columns wide. Only a few basic characters (Those used in the first 128 bits of a byte) are utilized in this stream, meaning that any terminal can display the text.

One of the nice things about it is that the encoding is predictable, basic and the text can be displayed on any terminal without sending evil escape characters and the like, which might make a login session go wonky. Unfortunately, it comes at the cost of size, as uuencoding increases the size of a binary file by about 35-40%. In the context of BatchLogin, this isn't an issue. It's not like BatchLogin's intent is to be efficient, rather the intent is to get it done without passing more credentials through an insecure system, or relying upon subsystems that have been turned or firewalled off.

Pushing Files

The basic methods are as follows: For pushing a file to a remote system, BatchLogin uuencodes the file into a text file, fires up uudecode on the remote machine via a "send" call (specifying the output file, of course), opens the local file with TCL read commands, and acts like an extremely tedious typewriter after that, sending each line of text as if it were a stenographer (albeit, an extremely fast one).

The remote system reads each line coming in from "standard input", which is in fact being controlled by expect. Since one in ever 1280 lines of input text sometimes ends up with an extra carriage return (certainly caused by some kind of terminal madness), the input is filtered first through "awk", which simply checks to see if the line is blank before passing it along.

Expect then ends the transmission, blows away the local uuencoded file (it's no longer needed) and does a checksum on both ends to make sure that nothing got toasted during transmission. If the sums don't match exactly, the remote file is blown away and expect signals the user that things didn't go as planned.

Pulling Files

You would think (and would be wrong) that this is exactly the same method employed to pull files from a remote host. Alas, the buffering problems on the remote system sending to the expect program are for some reason, worse. To remedy the situation, G. P. (bless his heart) came up with a method employing the "ex" editor. The ex editor is the precursor to vi, which still ships with all Unicies under the guise that someone might have a need for editing a file with a view that would make a world-war II tank operator squint. But I digress.

For pulling files, the ex editor is called upon to edit the file after it has been uuencoded on the remote host. Then, one line at a time, the file is displayed, and the output is buffered into a local file, which is run through dos2unix, and finally into the uudecode command on the local system. The label for the remote system, from which point this file transfer nonsense was executed, is affixed to the final output file.

At this point, both files are checksummed, and the resulting numbers compared. If the checksums differ, the file did not come down OK, and it is removed and the user signaled that something didn't work.

Space considerations

Care should be taken not to attempt large transport operations (keep it to at most, a few megabytes). Possibly, when things are in doubt, a small survey script that does some disk checking should be executed first. bl2 last (the command that edits the log file from the last run, quickly) is your friend.

[Back]