Using autossh

Background

Chartio recommends AutoSSH for all datasources connected via SSH tunnel.  Previously, Chartio distributed an application (Chartio Connect Client) which served an the same purpose, but this tool has been deprecated in favor of the open source AutoSSH.  All existing Connection Client connections will continue to operate, though it may be beneficial to switch. A user-managed SSH tunnel with AutoSSH provides more control over the connection which should make debugging much easier.

These instructions are specifically for setting up autossh on a Ubuntu/Debian server. For other operating systems, adjust the syntax as needed.

Note: these instructions are only applicable to existing connections that already use the Connection Client. For new connections, please see our documentation here.

Set-up instructions for Ubuntu/Debian

Install autossh

On your database server or jump host run install autossh:

For Ubuntu or Debian:

sudo apt-get install autossh

For Redhat, CentOS or Amazon Linux:

sudo yum install wget gcc make
wget http://www.harding.motd.ca/autossh/autossh-1.4e.tgz
tar -xf autossh-1.4e.tgz
cd autossh-1.4e
./configure
make
sudo make install

Verify autossh >= 1.4c is correctly installed

autossh -V

Get chartio_connect connection string

Run the following to view all open connections:

ps aux | grep ssh

Copy the SSH connection string for the specific connection client instance you want to update, and paste it into a text editor - you will need to reference this connection string soon. For example, ours looks like this:

ssh -N -R 12345:127.0.0.1:5432 tunnel12345@connect.chartio.com -g -i /home/chartio/.chartio.d/sshkey/id_rsa -p 22 -o ServerAliveInterval=10 -o ServerAliveCountMax=1 -o UserKnownHostsFile=/home/chartio/.chartio.d/sshkey/known_hosts

Kill chartio_connect instance

Run the following to get the location of your chartio_connect configuration:

crontab -l

Copy the command, without '@reboot', paste it into your terminal window, and append 'stop' to the end of the command before you run it. It should look something like this:

/usr/local/bin/chartio_connect -d --prefix=/home/chartio/instance stop

Create and run autossh connection string

Find your chartio_connect connection SSH string you saved earlier. Replace "ssh" with "autossh -M 0 -f". Your connection string should look something like this:

autossh -M 0 -f -N -R 12345:127.0.0.1:5432 tunnel12345@connect.chartio.com -g -i /home/chartio/.chartio.d/sshkey/id_rsa -p 22 -o ServerAliveInterval=10 -o ServerAliveCountMax=1 -o UserKnownHostsFile=/home/chartio/.chartio.d/sshkey/known_hosts

Create crontab entry

Once you've confirmed that the connection is up and running, you'll want to add a crontab entry to reconnect the SSH tunnel on reboot.

Additional considerations

Appending "-M 0" disables the monitoring port by default. From autossh's documentation:

If you are using a recent version of OpenSSH, you may wish to explore using the ServerAliveInterval and ServerAliveCountMax options to have the SSH client exit if it finds itself no longer connected to the server. In many ways this may be a better solution than the monitoring port.

Official definitions of ServerAliveInterval and ServerAliveCountMax, from autossh's documentation:

ServerAliveInterval
Sets a timeout interval in seconds after which if no data has been received from the server, ssh will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server, or 300 if the BatchMode option is set. This option applies to protocol version 2 only. ProtocolKeepAlives and SetupTimeOut are Debian-specific compatibility aliases for this option.

ServerAliveCountMax
Sets the number of server alive messages which may be sent without ssh receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is important to note that the use of server alive messages is very different from TCPKeepAlive. The server alive messages are sent through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive.

Additional connection string settings, such as logging, are available. See autossh documentation for details: http://www.harding.motd.ca/autossh/README

Troubleshooting

Ensure you have added a crontab entry for the connection

Confirm you have added a crontab entry to restart the connection on reboot.

Ensure the SSH process is not running

Run:

ps aux | grep ssh

If you see an SSH connection string for the connection you have switched to autossh, kill it using the instructions above.

Start autossh connection in the foreground with debugging enabled

To do this, remove '-f' from your autossh connection string and add AUTOSSH_DEBUG=1 to the beginning. Our example string now looks like this:

AUTOSSH_DEBUG=1 autossh -M 0 -N -R 12345:127.0.0.1:5432 tunnel12345@connect.chartio.com -g -i /home/chartio/.chartio.d/sshkey/id_rsa -p 22 -o ServerAliveInterval=10 -o ServerAliveCountMax=1 -o UserKnownHostsFile=/home/chartio/.chartio.d/sshkey/known_hosts

This will output the connection log, including any connection errors. If you do not understand the error messages, please send the output to support@chartio.com.