Interactive jobs using rrs
From EGEE-see WIki
Contents |
Introduction
Although an interactive connection with the running job is usually not needed, there are some cases when it can be extremely useful. These are often the cases when something has gones wrong, either with the grid software installed on the site or with the user application itself. In these cases an interactive shell connection to the problematic WN can be invaluable.
In order to get an interactive shell to a job running on a WN it is possible to use standard reverse shell techniques. Reverse shell is a similar concept to standard ssh access, but with the direction of the network connection reversed. In the ssh scenario, client connect to the server listening on a predefined port. In the case of reverse shell, client listens on a predefined port and server connects to it. Server here represents the remote computer running login shell (and user job) and client represents local computer from which the user is accessing the network. Reverse connection is needed because most sites restrict inbound access either by setting up a firewall or using private IP addresses for WNs.
Usage
Although any reverse shell can be used for this purpose, following examples will focus on Cycom Reverse Remote Shell. Before sending the job rrs client needs to be started on some predefined port. In the following example rrs starts listening for connections on TCP port 2000 on the local machine:
$ ./rrs -l -p 2000 [i] using plain-text communication [+] listening for incoming connection on port 2000, no timeout [i] got connection from 161.53.22.85:59265 seegrid035@node2:~$ id uid=3285(seegrid035) gid=1400(seegrid) groups=1400(seegrid) seegrid035@node2:~$ hostname node2 seegrid035@node2:~$ exit
When the connection from the job arrives the user can work on the remote host just like over ssh connection. Since the remote end has a terminal allocated even graphical applications (e.g. vi, top) can be used.
On the WN, rrs binary is executed as part of the job giving the hostname and port number as parameters:
$ globus-job-run grid1.irb.hr/jobmanager-pbs -q grid -s ./rrs home.irb.hr 2000 [i] using plain-text communication [+] connecting to home.irb.hr:2000 (161.53.22.20), using kernel connect() timeout [+] serving shell to home.irb.hr:2000 (161.53.22.20) [i] connection closed to home.irb.hr:2000 (161.53.22.20)
This examples shows the basic functionality of the remote shell. Other features also available in rrs but not described here are:
- reconnect,
- daemon mode, and
- different encryption algorithms including those available in OpenSSL.
See the rrs help for more information.
Conclusion
rrs is easy to use and fulfills most of the needs for reverse shell usage, but also leaves something to wish for. rrs is a C program that compiles into a dynamically linked binary and this can cause portability problems (libraries not available, incompatible architecture). This can partially be solved by using static compilation, but the problem of diverse architectures and operating systems still remains. And also some of the advanced options available in ssh are not present, most notably file transfer and port forwarding.
