AWS SSM Session Manager as secure Hub to establish multiple Connections to instances and Port Forwarding

Posted Feb 11, 2024 Updated Feb 26, 2024

By Roman Tsypuk 6 min read

Abstract

Port forwarding or SSH tunnelling is technic widely used to map remote ports to a client machine with possibility of remapping. The data traffic is encrypted and transferred within SSH tunnel. Such setups are usually used to establish connections to development environment or for fast troubleshooting.

Existing Implementations

Very often customers setup and configure ssh servers, maintain keys with the risk of leak and exposure, rotation, etc. Also in many AWS learning materials and certifications there is a technic called the Jump Box or a Bastion Host - the machine that is publicly available for incoming connection that is the entry point for admin.

For ECS containers, some sysops are even prebuild image with ssh server installed with keys.

But all this technics besides human factor risks, are exposing opened SSH port, making the machines first of all discoverable and potentially vulnerable for the attacks.

In this post we will explore a more secure and reliable approach that AWS Systems Manager provides - the Session Manager feature. It allows to establish secure connection to running instances without exposed ports or installing ssh servers. AWS Systems Manager once configuration is applied mounts external ephemeral storage that has preinstalled agent, keys and configuration. Also on top this setup is fully controlled by AWS IAM.

Network configuration

Using System Manager Session Manager with Port Forwarding mode does not require you to apply any updates in Security Groups, Network Access Control Lists, VPC routing, etc. The Session established from you local client till remote server agent is encrypted and traffic or port forwarding runs inside the tunnel. This is excellent tool for urgent troubleshooting remote instances that are not exposed and stay in private network (no need to break network setup).

A session is a connection made to a managed node using Session Manager. Sessions are based on a secure bi-directional communication channel between the client (you) and the remote managed node that streams inputs and outputs for commands. Traffic between a client and a managed node is encrypted using TLS 1.2, and requests to create the connection are signed using Sigv4. This two-way communication allows interactive bash and PowerShell access to managed nodes. You can also use an AWS Key Management Service (AWS KMS) key to further encrypt data beyond the default TLS encryption.

Client-side configuration

There are prerequisites that require both client and remote side configurations to start using SSM. Full configuration with troubleshooting steps is described in previous post: Securely accessing ECS Fargate containers shell, without SSH or exposed ports.

In this post we will concentrate on real practical use cases.

Connecting to remote ECS container

To connect shell of running ECS task - we need to specify following params:

Param	Value
cluster	fargate-cluster
taskId	9cdef64a51841c93c26d5
containerId	9cdef64a51841c93c26d40abd5-5245480

  
aws ssm start-session \
--target ecs:fargate-cluster_9cdef64a51841c93c26d5_9cdef64a51841c93c26d40abd5-5245480 \

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.
Starting session with SessionId:
bash-5.0#

Connection to remote EC2 instance

Connection to EC2 instance can be done either through aws cloud console or running locally:

Param	Value
target	EC2 instance ID

  
aws ssm start-session --region eu-west-1 \
--target instanceId

SSM PARAM: document-name

If we check the documentation there is additional parameter called document-name that allows to specify aws ssm what additional actions should connection perform. One of the most common used is port forwarding from remote to local machine. Let’s explore in details.

Port-forwarding of ECS container

Connect 1880 remote ECS container port to 1880 local machine port.

Param	Value
cluster	fargate-cluster
taskId	9cdef64a51841c93c26d5
containerId	9cdef64a51841c93c26d40abd5-5245480

  
aws ssm start-session \
--target ecs:fargate-cluster_9cdef64a51841c93c26d5_9cdef64a51841c93c26d40abd5-5245480 \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["1880"], "localPortNumber":["1880"]}'

Starting session with SessionId: xxxx
Port 1880 opened for sessionId xxx
Waiting for connections...

Connection accepted for session [xxx]

Running this command on your local machine:

a secure tunnel direct connection is established till 9cdef64a51841c93c26d40abd5 container
client machine 1880 port is mapped to 1880 port of remote container port through secure tunnel

In this example we are connecting to remote NodeRed application that is by default running on 1880 port, but without opening the port.

Port-forwarding of EC2

Connect 80 remote EC2 instance port to 8080 local machine port.

Param	Value
target	EC2 instance ID

  
aws ssm start-session \
--target EC2-instance-ID \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["80"], "localPortNumber":["8080"]}'

AWS-StartPortForwardingSessionToRemoteHost

And even more :). With AWS-StartPortForwardingSessionToRemoteHost we can connect to JumpBox instance and run further 2nd hop port forwarding to RDS instance: Connecting to RDS through EC2 JumpBox of the same subnet:

Param	Value
target	EC2 instance ID
host	RDS
portNumber	RDS port
localPortNumber	local machine 2nd hop port

  
aws ssm start-session --region eu-west-1 \
--target <your jump host instance id> \
--document-name AWS-StartPortForwardingSessionToRemoteHost \
--parameters host="<rds endpoint>",portNumber="3306",localPortNumber="3306"

> mysql --host=127.0.0.1

Required IAM Role Permission

To allow SSM interact with a service - following Permissions must be added to existing IAM Role of ECS service:

  
 {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": "*"
    }
  ]
}

ECS Task enableExecuteCommand updates

In Task Definition "enableExecuteCommand": true, should be enabled. If we are planning to update existing Task following command can be executed (it will require redeployment of new service version):

Param	Value
cluster	fargate-cluster
service	service name

  
aws ecs update-service --cluster dev-fargate-cluster \
    --enable-execute-command --service xxxxxx-fg-svc \
    --force-new-deployment

ECS Task check applied updates

Param	Value
cluster	fargate-cluster
tasks	task ARN

aws ecs describe-tasks --cluster fargate-cluster --tasks arn:aws:ecs:eu-west-1:123456789012:task/fargate-cluster/xxxxxx

  
{
  "tasks": [
    {
      "attachments": [
        {
          "id": "b5018ca9-b683-4f9c-xxxxxx-xxxxxx",
          "type": "ElasticNetworkInterface",
          "status": "ATTACHED",
          "details": [
            {
              "name": "subnetId",
              "value": "subnet-xxxxxx"
            },
            {
              "name": "networkInterfaceId",
              "value": "eni-xxxxxx"
            },
            {
              "name": "macAddress",
              "value": "0a:b8:xx:xx:0c:df"
            },
            {
              "name": "privateDnsName",
              "value": "ip-10-xx-2-xx.eu-west-1.compute.internal"
            },
            {
              "name": "privateIPv4Address",
              "value": "10.xx.2.xx"
            }
          ]
        }
      ],
      "attributes": [
        {
          "name": "ecs.cpu-architecture",
          "value": "x86_64"
        }
      ],
      "availabilityZone": "eu-west-1b",
      "clusterArn": "arn:aws:ecs:eu-west-1:123456789012:cluster/fargate-cluster",
      "connectivity": "CONNECTED",
      "connectivityAt": 1708944472.521,
      "containers": [
        {
          "containerArn": "arn:aws:ecs:eu-west-1:123456789012:container/fargate-cluster/xxxxxx/xxxxxx",
          "taskArn": "arn:aws:ecs:eu-west-1:123456789012:task/fargate-cluster/xxxxxx",
          "name": "xxxxxx",
          "image": "123456789012.dkr.ecr.eu-west-1.amazonaws.com/xxxxxx/xxxxxx:xxxxxx",
          "imageDigest": "sha256:xxxxxx",
          "runtimeId": "xxxxxx-xxxxxx",
          "lastStatus": "RUNNING",
          "networkBindings": [],
          "networkInterfaces": [
            {
              "attachmentId": "xxxxxx-b683-4f9c-b521-xxxxxx",
              "privateIpv4Address": "10.0.xxxxxx.xxxxxx"
            }
          ],
          "healthStatus": "HEALTHY",
          "managedAgents": [
            {
              "lastStartedAt": 1708944503.722,
              "name": "ExecuteCommandAgent",
              "lastStatus": "RUNNING"
            }
          ],
          "cpu": "256",
          "memory": "512"
        }
      ],
      "cpu": "256",
      "createdAt": 1708944469.064,
      "desiredStatus": "RUNNING",
      "enableExecuteCommand": true,
      "group": "service:xxxxxx-fg-svc",
      "healthStatus": "HEALTHY",
      "lastStatus": "RUNNING",
      "launchType": "FARGATE",
      "memory": "512",
      "overrides": {
        "containerOverrides": [
          {
            "name": "xxxxxx"
          }
        ],
        "inferenceAcceleratorOverrides": []
      },
      "platformVersion": "1.4.0",
      "platformFamily": "Linux",
      "pullStartedAt": 1708944483.325,
      "pullStoppedAt": 1708944500.38,
      "startedAt": 1708944509.874,
      "startedBy": "ecs-svc/xxxxxx",
      "tags": [],
      "taskArn": "arn:aws:ecs:eu-west-1:123456789012:task/fargate-cluster/xxxxxx",
      "taskDefinitionArn": "arn:aws:ecs:eu-west-1:123456789012:task-definition/xxxxxx-fg-task:16",
      "version": 5,
      "ephemeralStorage": {
        "sizeInGiB": 20
      }
    }
  ],
  "failures": []
}

References (Links)

aws, ssm, ssh, security

This post is licensed under CC BY 4.0 by the author.