AWS VPC traffic mirroring to IDS. Deep dive with Wireshark.

Abstract

Traffic mirroring refers to the process of copying network traffic from one network port or interface to another for analysis, monitoring, or security purposes. There are multiple options for traffic mirroring depending on what hardware/virtual setup is in use. The most popular modes of interaction with the system during traffic inspection are:

• IDS (Intrusion Detection System) - This system detects suspicious network behavior and generates alerts or notifications if any rules are violated, such as unexpected source/port/protocol. The rules are declared in predefined format (we will check the popular Suricata-based format of rules).
• IPS (Intrusion Prevention System) - This system goes a step further than IDS and not only detects and alerts about suspicious behavior, but also has the capability to influence the traffic, such as dropping packets or changing values.

AWS offers multiple options to achieve these and probably the top is a managed AWS Network Firewall service that operates as an IPS allowing for traffic inspection and application of different rules. This solution is built on the open-source Suricata system. Also, there is option with Transient Gateway and Centralized Egress Appliance VPC but here you should be careful in setup not to have asymmetric traffic drops. Since 2020 newly introduced AWS Gateway Load Balancer can cover IPS scenario.

Here, we will take a look at option with VPC traffic mirroring to check how it works and what happens with packets.

Network setup for traffic mirroring

In current setup we will configure VPC traffic mirroring and will use Suricata to inspect packets. You can use any other IDS/IPS - personally I more like Snort, but for this setup Suricata will be used.

To make it easy compared captured packets we will apply filter on top of traffic mirroring session to mirror only incoming and outgoing IMCP ping packets.

To start traffic flow on the source origin instance, we will run ping command to one of Google’s public IPs.

ping 142.251.42.79

Configuring VPC traffic mirroring

If you are from Cisco world, the setup process is very similar to configuring port span on equipment. We need to follow step of mapping the target - this can be ENI, NLB or Gateway LB endpoint:

We have control over Inbound and Outbound Rules in traffic filter. If you are interested in specific data traffic that can be classified by source/destinatio CIDRs and ports, by applying them in these filters you will mirror only traffic that is compatible with filter.

And do not forget to start mirror session. Here we specify the traffic source - the ENI of our application instance, target (previously created) and VNI. If VNI is not specified it will be randomly generated. But here we will explicitly define it with value 333, so later in captured traffic we will track this value.

Suricata installation

Next, lets login into appliance instance, install and configure Suricata. Here we are configuring Suricata rules:

• will alert for ICMP from/to any src/dst with message “ICMP traffic detected”

Later we will check why UDP and 4789 port are used.

sudo -s
# Install epel-release
amazon-linux-extras install -y epel
# Install suricata
yum install -y suricata
# Create the default suricata rules directory
mkdir /var/lib/suricata/rules
# Add a rule to match all UDP traffic
echo 'alert icmp any any -> any any (msg:"ICMP traffic detected"; sid:200001; rev:1;)' > /var/lib/suricata/rules/suricata.rules
# Start suricata listening on eth0 in daemon mode
suricata -c /etc/suricata/suricata.yaml -k none -i eth0 -D
# Capture logs can be found in /var/log/suricata/fast.log
yum install -y jq
tail -f /var/log/suricata/eve.json  | jq 'select(.event_type=="alert")'

Let’s disable Jumbo-frames for this setup:

sudo ip link set dev eth0 mtu 1500

NW interfaces on application server

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.31.9.144  netmask 255.255.240.0  broadcast 172.31.15.255
        inet6 fe80::8d3:93ff:fee4:27b2  prefixlen 64  scopeid 0x20<link>
        ether 0a:d3:93:e4:27:b2  txqueuelen 1000  (Ethernet)
        RX packets 67739  bytes 91125253 (86.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8329  bytes 777401 (759.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

NW interfaces Suricata appliance

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.31.4.131  netmask 255.255.240.0  broadcast 172.31.15.255
        inet6 fe80::836:d2ff:fefc:942e  prefixlen 64  scopeid 0x20<link>
        ether 0a:36:d2:fc:94:2e  txqueuelen 1000  (Ethernet)
        RX packets 147218  bytes 194694578 (185.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22560  bytes 2819722 (2.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Running a ping from application server

First lets monitor Suricata logs:

[root@ip-172-31-4-131 ec2-user]# tail -f /var/log/suricata/fast.log

Running ping from application server to public IP - there are no new records in Suricata log.

But when are trying to directly ping Suricata appliance from Application server we see new entries:

[root@ip-172-31-4-131 ec2-user]# tail -f /var/log/suricata/fast.log
02/08/2023-XX:19:07.005845  [**] [1:200001:1] ICMP traffic detected [**] [Classification: (null)] [Priority: 3] {ICMP} 172.31.9.144:3 -> 172.31.4.131:3
02/08/2023-XX:19:07.006011  [**] [1:200001:1] ICMP traffic detected [**] [Classification: (null)] [Priority: 3] {ICMP} 172.31.4.131:3 -> 172.31.9.144:3

Buy the way, Suricata also supports logging in JSON format, very useful to query using JQ utility, we can see same entries but with more details:

tail -f /var/log/suricata/eve.json  | jq 'select(.event_type=="alert")'
{
  "timestamp": "2023-02-08TXX:56:10.323036+0000",
  "flow_id": 1671295421574620,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "172.31.9.144",
  "dest_ip": "172.31.4.131",
  "proto": "ICMP",
  "icmp_type": 8,
  "icmp_code": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 200001,
    "rev": 1,
    "signature": "ICMP traffic detected",
    "category": "",
    "severity": 3
  },
  "flow": {
    "pkts_toserver": 1,
    "pkts_toclient": 0,
    "bytes_toserver": 98,
    "bytes_toclient": 0,
    "start": "2023-02-08TXX:56:10.323036+0000"
  }
}
{
  "timestamp": "2023-02-08TXX:56:10.323195+0000",
  "flow_id": 1671295421574620,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "172.31.4.131",
  "dest_ip": "172.31.9.144",
  "proto": "ICMP",
  "icmp_type": 0,
  "icmp_code": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 200001,
    "rev": 1,
    "signature": "ICMP traffic detected",
    "category": "",
    "severity": 3
  },
  "flow": {
    "pkts_toserver": 1,
    "pkts_toclient": 1,
    "bytes_toserver": 98,
    "bytes_toclient": 98,
    "start": "2023-02-08TXX:56:10.323036+0000"
  }
}

Capture and inspect traffic with Wireshark

Now lets capture ENIs of both application and appliance instances and compare them.

This is a capture of ping response from 142.251.42.79 to our 172.31.9.144 Application server.

As we can observe, the packets are different. Left side of the screen is a capture of Application server, right - Appliance instance. The initial source ICMP packet is wrapped by VPC traffic mirroring session by UDP packet with additional headers. Here we have capture same ICMP packet sequence number - you can see there is no difference in the inner encapsulated payload.

The outer packet has source of ENI that we are capturing and destination of ENI on target appliance. Also, this is UDP packet sent to 4789 port. That is why initially we did not see any records in log in ICMP ping commands. Also take a look at header VXLAN Network Identifier (VNI) it has value 333 that we have configured previously in session mirror setup.

On the right bottom section of the screen you can see how packet encapsulation is applied in a binary form.

UDP traffic mirroring logging:

If we update Suricata with new rule:

RULE: alert udp any any -> any 4789 (msg:"UDP Mirror traffic detected"; sid:200001; rev:1;)

Now we can see all mirrored UDP packet incoming into Appliance instance. But with one remark, that these are encapsulated UDP packets and the src/dst are not the same as in origin packet. So it is just for observation, but we can not create and apply any rule for them. We will fix it in a while.

tail -f /var/log/suricata/eve.json  | jq 'select(.event_type=="alert")'
{
  "timestamp": "2023-01-27T13:10:48.181155+0000",
  "flow_id": 1642158420968355,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "172.31.9.144",
  "src_port": 65511,
  "dest_ip": "172.31.4.131",
  "dest_port": 4789,
  "proto": "UDP",
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 200001,
    "rev": 1,
    "signature": "UDP Mirror traffic detected",
    "category": "",
    "severity": 3
  },
  "app_proto": "failed",
  "flow": {
    "pkts_toserver": 1,
    "pkts_toclient": 0,
    "bytes_toserver": 148,
    "bytes_toclient": 0,
    "start": "2023-01-27T13:10:48.181155+0000"
  }
}

This is actually how VXLan works

VXLAN (Virtual Extensible LAN) is a network virtualization technology that uses a VLAN-like encapsulation technique to encapsulate MAC-based OSI Layer 2 Ethernet frames within Layer 3 IP packets. This allows communication between virtualized network segments, providing network isolation and enables the creation of multiple virtual LANs on a single physical infrastructure.

Here’s a summary of how VXLAN works:

• Endpoints (VMs or physical hosts) communicate with each other as if they were on the same LAN, despite being located in different physical locations.
• The VXLAN encapsulation process adds a VXLAN header to the original Ethernet frame, including a 24-bit VXLAN Network Identifier (VNI), which acts as a segment ID for the virtual network.
• The encapsulated frame is then sent as a standard IP packet to the destination endpoint.
• The destination endpoint removes the VXLAN header and processes the original Ethernet frame as if it had never left the local network.

Adding vxlan interface

Usually appliance instance can operate with vxlan traffic and extract encapsulated inner packages. To achieve this - we can add virtual interface on top of eth0 that will perform these actions.

sudo ip link add vxlan100 type vxlan id 333 dstport 4789 local 172.31.4.131 dev eth0
sudo ip link set vxlan100 up

Suricata appliance is now showing second interface:

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.31.4.131  netmask 255.255.240.0  broadcast 172.31.15.255
        inet6 fe80::836:d2ff:fefc:942e  prefixlen 64  scopeid 0x20<link>
        ether 0a:36:d2:fc:94:2e  txqueuelen 1000  (Ethernet)
        RX packets 147579  bytes 194728353 (185.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22902  bytes 2910203 (2.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vxlan100: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::ecb1:f6ff:fe9a:c044  prefixlen 64  scopeid 0x20<link>
        ether ee:b1:f6:9a:c0:44  txqueuelen 1000  (Ethernet)
        RX packets 2  bytes 168 (168.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

And now if capture Suricata to vxlan100 port - we will see original packet data, src/dst/proto to which we can apply proper rules:

[root@ip-172-31-4-131 ec2-user]# suricata -c /etc/suricata/suricata.yaml -k none -i vxlan100: -D
[root@ip-172-31-4-131 ec2-user]# tail -f /var/log/suricata/eve.json  | jq 'select(.event_type=="alert")'
{
  "timestamp": "2023-02-08TXX:59:54.986033+0000",
  "flow_id": 650854156340145,
  "in_iface": "vxlan100:",
  "event_type": "alert",
  "src_ip": "172.31.9.144",
  "dest_ip": "142.251.42.79",
  "proto": "ICMP",
  "icmp_type": 8,
  "icmp_code": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 200001,
    "rev": 1,
    "signature": "ICMP traffic detected",
    "category": "",
    "severity": 3
  },
  "flow": {
    "pkts_toserver": 1,
    "pkts_toclient": 0,
    "bytes_toserver": 98,
    "bytes_toclient": 0,
    "start": "2023-02-08TXX:59:54.986033+0000"
  }
}
{
  "timestamp": "2023-02-08TXX:59:54.987442+0000",
  "flow_id": 650854156340145,
  "in_iface": "vxlan100:",
  "event_type": "alert",
  "src_ip": "142.251.42.79",
  "dest_ip": "172.31.9.144",
  "proto": "ICMP",
  "icmp_type": 0,
  "icmp_code": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 200001,
    "rev": 1,
    "signature": "ICMP traffic detected",
    "category": "",
    "severity": 3
  },
  "flow": {
    "pkts_toserver": 1,
    "pkts_toclient": 1,
    "bytes_toserver": 98,
    "bytes_toclient": 98,
    "start": "2023-02-08TXX:59:54.986033+0000"
  }
}

Wireshark capture on vxlan100

Let’s capture traffic again on both instances, but this time on appliance instace we will use vxlan port.

vxlan interface capture (filtered ICMP packets only):

Compare both captures - the data is the same (now there is no encapsulation with UDP, and we can see native raw packets as on the origin).

Conclusions

AWS VPC traffic mirroring is a powerful tool that can help improve the security of a network by allowing network administrators to monitor and inspect all incoming and outgoing traffic in real-time. This feature can be particularly useful when used in conjunction with an Intrusion Detection System (IDS) as it provides valuable insights into network activity that can be used to detect and prevent malicious activity.

Links

• AWS Network Firewall
• Suricata rules format
• AWS VPC traffic mirroring