• AWS SSM HTTPS/SSH Reverse Tunnel

    23 September 2018
    Tags: cloud 

    The Amazon AWS Systems Manager Agent (SSM Agent) is a great way to manage systems in EC2 or on premises. It can run shell commands remotely and return a response. But sometimes there is no substitute for a full SSH session. The problem is most firewalls will block incoming or outgoing SSH connections because of the security risk. This guide shows a simple way to trigger a reverse tunnel with SSH over HTTPS back to an EC2 instance you can use to remotely control a system. All SSH keys are generated on demand and never reused. The remote connection will use a service account that does not provide a shell. Keys are removed after the connection is made so the tunnel cannot be easily hijacked.

    This guide assumes you are using Ubuntu Linux 16.04 LTS on the server and remote client.

    1. Set up an EC2 Apache2 server for SSH over HTTPS

    First you will need something to connect back to. You could use a plain reverse SSH tunnel but most corporate firewalls will stop this.

    The best solution is to install Apache2 and forward incoming HTTPS traffic to a local SSH server.

    Spin up a new Ubuntu 16.04 EC2 instance with a public IP and connect to it with SSH. Be sure to allow HTTPS (port 443) traffic in from the public Internet. Also set up a public DNS name for your HTTPS certificates.

    Install Apache2 and activate the modules you need for HTTPS and proxying.

    sudo apt-get install apache2
    sudo a2enmod ssl
    sudo a2enmod proxy
    sudo a2enmod proxy_http

    You can provision certificates with LetsEncrypt or install your own.

    Update your Apache2 configuration using the example below. Be sure to change the following lines to point to your SSL certificates.

    SSLCertificateFile /etc/letsencrypt/live/your_site_name/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/your_site_name/privkey.pem

    <IfModule mod_ssl.c>
            <VirtualHost _default_:443>
                    ServerAdmin webmaster@localhost
                    DocumentRoot /var/www/html
                    ErrorLog ${APACHE_LOG_DIR}/error.log
                    CustomLog ${APACHE_LOG_DIR}/access.log combined
                    #   SSL Engine Switch:
                    #   Enable/Disable SSL for this virtual host.
                    SSLEngine on
                    SSLCertificateFile      /etc/letsencrypt/live/your_site_name/fullchain.pem
                    SSLCertificateKeyFile /etc/letsencrypt/live/your_site_name/privkey.pem
                    <FilesMatch "\.(cgi|shtml|phtml|php)$">
                                    SSLOptions +StdEnvVars
                    <Directory /usr/lib/cgi-bin>
                                    SSLOptions +StdEnvVars
            # proxytunnel
            ProxyRequests On
            AllowConnect 22
            <Proxy *>
                # Deny all proxying by default ...
                Require all denied
                # Now allow proxying by localhost only
                Require all granted

    There are lots of good guides on how to get this working if you get stuck. Just Google it: https://www.google.co.uk/search?q=ssh+over+https+apache2

    2. Setup this script on your EC2 server

    This script will automate the process of creating a new SSH keypair and finding a free local port to connect to. It will also trigger the SSM commands and setup the reverse tunnel for you.

    Setup / configure awscli on your Ubuntu EC2 server and ensure it is up to date by running the following commands. Be sure to use an IAM service account for awscli configure that can access SSM to query instances and run commands.

    sudo apt-get awscli
    sudo pip3 install awscli --upgrade
    sudo aws configure

    Create a new local Linux user just for the SSH connection and remove their shell access. Be sure to set a complex password and harden your system accordingly. e.g. Install fail2ban

    sudo adduser sshconnectuser
    sudo usermod -s /bin/false sshconnectuser

    Create a new script named sshconnect-server.sh with nano. Make it executable with chmod +x sshconnect-server.sh. Be sure to replace YOUR_PUBLIC_DNS_NAME_HERE with your EC2 instance's public DNS name.

    # Configuration
    # SSM document name
    # remove any old keys
    sudo rm -f /dev/shm/id_rsa.pub
    sudo rm -f /dev/shm/id_rsa
    # generate a fresh key pair without a passphrase as sshconnectuser
    sudo -H -u sshconnectuser bash -c "ssh-keygen -N '' -b 2048 -f /dev/shm/id_rsa"
    # copy the key to the correct location for the sshconnect user
    sudo mkdir -p /home/sshconnectuser/.ssh/
    sudo cp /dev/shm/id_rsa.pub /home/sshconnectuser/.ssh/authorized_keys
    # sshconnectuser should own the file
    sudo chown sshconnectuser:sshconnectuser /home/sshconnectuser/.ssh/authorized_keys
    # Copy the private key to a variable
    SSH_KEY=$(sudo cat /dev/shm/id_rsa)
    # Obtain a list of SSM agents
    INSTANCE_LIST=$(aws ssm describe-instance-information --query "InstanceInformationList[*].[InstanceId, ComputerName]" --output text)
    # Select open port on localhost starting with 11111
    while true; do
      nc -zv ${SSH_PORT}
      if [ $? == 0 ]
        SSH_PORT=$(expr ${SSH_PORT} + 1)
    # Show a list of SSM instances to remotely access
    INSTANCE=$(whiptail --title "Choose a host" --menu "Instance ID, ComputerName" 24 50 14 ${INSTANCE_LIST} 3>&1 1>&2 2>&3)
    # Exit if menu canceled
    if [[ ${?} == 1 ]]
      exit 0
    # Run the SSM task to trigger the remote host's reverse tunnel, this references the SSM document on AWS and passes in the instance / port / key parameters
    aws ssm send-command --document-name ${SSM_DOCUMENT} --targets "Key=instanceids,Values=${INSTANCE}" \
    echo "Waiting 15 seconds for the connection"
    sleep 15
    # Connect to SSH reverse tunnel
    ssh -p ${SSH_PORT}

    3. Create a new YAML AWS SSM document

    Be sure to name it sshconnect-ssm so the script on your Ec2 instance can call it. More info about how to create a new SSM document: https://docs.aws.amazon.com/systems-manager/latest/userguide/create-ssm-doc.html

    This document will take the variables from your sshconnect-server script and execute the correct commands on the remote host.

    schemaVersion: "2.2"
    description: "Copy a private key to a remote host and establish a reverse tunnel on a specific port.  Document on AWS must be named: sshconnect-ssm"
        type: "String"
        description: "SSH private key"
        default: ""
        type: "String"
        description: "SSH server public DNS name"
        default: ""
        type: "String"
        description: "SSH username"
        default: "sshconnectuser"
        type: "String"
        description: "Remote port for SSH"
        default: ""
    - action: "aws:runShellScript"
      name: "keygen"
          - "rm -f /dev/shm/id_rsa"
          - "touch /dev/shm/id_rsa"
          - "chmod 400 /dev/shm/id_rsa"
          - "echo '' > /dev/shm/id_rsa"
          - "/usr/local/bin/sshconnect-client.sh   "
          - "rm -f /dev/shm/id_rsa"

    4. Setup SSM on your remote host

    This page explains how to deploy the agent. Be sure to register it after you deploy it and confirm it is checking in.

    Agent install for Ubuntu: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html#agent-install-ubuntu-deb

    Proxy configuration: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-proxy-with-ssm-agent.html

    5. Install ProxyTunnel and deploy this script on the remote machine

    Create the script in /usr/local/bin/ and name it sshconnect-client.sh. Ensure you make it executable with sudo chmod +x /usr/local/bin/sshconnect-client.sh. This script will handle the connection back to the EC2 server. It will also attempt to work out the local proxy if one is being used.

    # This script should reside on the client system, it is called remotely by AWS SSM and will connect back over HTTPS with SSH and setup a reverse tunnel
    TUNNEL_COMMAND_BASE="ProxyCommand proxytunnel -q -d"
    # Work out the correct proxy scenario
    # Is a local proxy between the HTTPS server and the client?
    if $(grep -q 'https_proxy' /etc/environment)
      # Find proxy string and remove prefix
      PROXY_STRING=$(grep https_proxy /etc/environment | sed 's/"//g' | cut -d "=" -f 2 | sed 's/^........//')
      # Add proxy auth to the tunnel command if required
      if [[ ${PROXY_STRING} = *"@"* ]]
        # Connect to authenticated proxy and then create an encrypted tunnel to HTTPS server
        # Environment variables must be used to support complex URL encoded passwords
        # Get only auth string
        PROXY_AUTH=$(echo ${PROXY_STRING} | cut -d "@" -f 1)
        # Get username
        set PROXYUSER=$(echo ${PROXY_AUTH} | cut -d ":" -f 1)
        # Get password
        set PROXYPASS=$(echo ${PROXY_AUTH} | cut -d ":" -f 2)
        PROXY_HOST=$(echo ${PROXY_STRING} | cut -d "@" -f 2)
      # Connect to local proxy
     # No proxy in between
    # Build tunnel command for correct scenario
    # Build the command
    SSH_COMMAND="ssh -f -q -v -N \
    -i /dev/shm/id_rsa \
    -o 'UserKnownHostsFile=/dev/null' \
    -o 'HostName ${SSH_HOST}' \
    -o 'Port 443' \
    -o 'StrictHostKeyChecking=no' \
    -o '${TUNNEL_COMMAND}' \
    -R${SSH_PORT_REMOTE}:localhost:22 ${SSH_USERNAME}@${SSH_HOST}"
    # Run the built command
    eval ${SSH_COMMAND}

    6. Run the server side script

    Call your sshconnect-server.sh script and it will prompt you for available SSM instances. Then select one that you deployed the client script to, it should connect back to your server over HTTPS and use the local proxy if one is required. You can then login with the username and password of the remote system.