Introduction: The Importance of Layer-by-Layer Security

To properly understand network security, you first need to know how networks work. The OSI 7-layer model and TCP/IP model are core frameworks for understanding network communication, and each layer has its own unique security threats and defense methods.

In this article, we'll review the OSI model and cover representative attack techniques at each layer along with countermeasures. We'll also look at how to use Wireshark, an essential tool for analyzing actual network traffic.

1. OSI 7-Layer Model Review

The OSI (Open Systems Interconnection) model is a reference model that explains network communication divided into 7 layers.

Layer Name Main Function Protocols/Devices PDU
7 Application User interface, application services HTTP, FTP, SMTP, DNS Data
6 Presentation Data format conversion, encryption, compression SSL/TLS, JPEG, ASCII Data
5 Session Session setup, maintenance, termination NetBIOS, RPC Data
4 Transport End-to-end communication, flow control, error recovery TCP, UDP Segment
3 Network Logical addressing, routing IP, ICMP, Router Packet
2 Data Link Physical addressing, frame creation, error detection Ethernet, ARP, Switch Frame
1 Physical Bit transmission, physical connection Cables, Hub, Repeater Bit

OSI vs TCP/IP Model Comparison

OSI 7-Layer          TCP/IP 4-Layer
+------------+     +------------+
| Application|     |            |
+------------+     | Application|
| Presentation|    |            |
+------------+     +------------+
| Session    |     |            |
+------------+     +------------+
| Transport  |     | Transport  |
+------------+     +------------+
| Network    |     | Internet   |
+------------+     +------------+
| Data Link  |     |            |
+------------+     | Network    |
| Physical   |     | Interface  |
+------------+     +------------+

2. Physical Layer Security (Layer 1)

The physical layer is the most fundamental layer of a network, dealing with actual hardware and electrical signals.

2.1 Physical Layer Threats

  • Wiretapping: Physically accessing cables to intercept signals
  • Equipment theft/damage: Physical theft or destruction of servers, routers, switches
  • Electromagnetic Interference (EMI): Interfering with or eavesdropping on cable signals
  • Wireless eavesdropping: Collecting and analyzing wireless signals

2.2 Physical Layer Security Measures

  • Physical access control: Server room locks, access cards, CCTV installation
  • Cable protection: Shielded cables (STP), cable duct installation
  • Fiber optic use: Resistant to EMI and difficult to tap
  • Environmental monitoring: Temperature, humidity, flood detection sensors
  • Wireless security: WPA3 encryption, hidden SSID, MAC filtering

3. Data Link Layer Security (Layer 2)

The data link layer handles communication between nodes within the same network segment. It uses MAC addresses, and switches operate at this layer.

3.1 ARP Spoofing

ARP (Address Resolution Protocol) converts IP addresses to MAC addresses. ARP spoofing exploits vulnerabilities in this protocol.

Attack Principle

Normal ARP Operation:
PC-A: "What's the MAC address for 192.168.1.1?" (ARP Request - Broadcast)
Gateway: "My MAC address is AA:BB:CC:DD:EE:FF" (ARP Reply)

ARP Spoofing Attack:
Attacker: "The MAC address for 192.168.1.1 is my MAC (XX:XX:XX:XX:XX:XX)!" (Forged ARP Reply)
PC-A: Mistakes attacker's MAC for gateway → All traffic sent to attacker

Attack Results

  • Man-in-the-Middle (MITM) attacks possible
  • Session hijacking
  • Traffic eavesdropping and modification
  • Denial of Service (DoS)

Countermeasures

  • Static ARP tables: Manually fix MAC addresses for critical devices
  • Dynamic ARP Inspection (DAI): Validate ARP packets at the switch
  • ARP monitoring tools: Monitor ARP table changes with arpwatch, XArp, etc.
  • Encrypted communication: Protect content even when eavesdropped using HTTPS, VPN

3.2 MAC Flooding

An attack that fills a switch's MAC address table (CAM table) with fake MAC addresses to disrupt normal operation.

Attack Principle

Normal Switch Operation:
- Forwards frames only to specific ports based on learned MAC addresses in MAC table
- Efficient network operation with unicast communication

During MAC Flooding Attack:
1. Attacker sends thousands to tens of thousands of fake MAC addresses
2. Switch's CAM table becomes full (typically 8K-32K entries)
3. Switch operates like a hub (broadcasts frames to all ports)
4. Attacker can receive all network traffic

Countermeasures

  • Port Security: Limit allowed MAC addresses per port
  • 802.1X Authentication: Authenticate users/devices before network access
  • VLAN Segmentation: Separate broadcast domains
# Cisco switch port security configuration example
Switch(config)# interface FastEthernet0/1
Switch(config-if)# switchport mode access
Switch(config-if)# switchport port-security
Switch(config-if)# switchport port-security maximum 2
Switch(config-if)# switchport port-security violation shutdown
Switch(config-if)# switchport port-security mac-address sticky

3.3 VLAN Hopping

An attack for unauthorized access to other VLANs, using two methods: switch spoofing and double tagging.

  • Switch Spoofing: Attacker pretends to be a switch to establish trunk port connection
  • Double Tagging: Using two VLAN tags to send packets to another VLAN

Countermeasures

  • Disable unused ports
  • Disable DTP (Dynamic Trunking Protocol) on access ports
  • Change native VLAN to an unused VLAN
  • Explicitly configure trunk ports

4. Network Layer Security (Layer 3)

The network layer handles logical addressing and routing using IP addresses.

4.1 IP Spoofing

An attack that forges source IP addresses to hide identity or impersonate another host.

Attack Purposes

  • Anonymity: Concealing attack origin
  • Authentication bypass: Bypassing IP-based access controls
  • DDoS attacks: Impersonating victim's IP in reflection/amplification attacks
  • Session hijacking: Taking over existing connections
IP Spoofing Example (DDoS Reflection Attack):

1. Attacker → DNS Server (Source: Victim IP, Destination: DNS Server)
   "DNS query request"

2. DNS Server → Victim (Source: DNS Server, Destination: Victim IP)
   "DNS response (much larger than request)"

Result: Victim receives massive unsolicited response traffic

Countermeasures

  • Ingress/Egress Filtering: Block packets with abnormal source IPs
  • BCP38/BCP84: Apply spoofing prevention at ISP level
  • uRPF (Unicast Reverse Path Forwarding): Verify source IP at router
  • Cryptographic authentication: Authenticate using cryptographic methods instead of IP addresses

4.2 ICMP Attacks

ICMP (Internet Control Message Protocol) is used for network diagnostics and error reporting but can be exploited for attacks.

Major ICMP Attack Types

Attack Type Description Impact
Ping Flood Sending large amounts of ICMP Echo Requests Bandwidth exhaustion, DoS
Ping of Death Sending abnormally large ICMP packets System crash (older systems)
Smurf Attack Spoofed ping to broadcast address Amplified DDoS
ICMP Redirect Injecting malicious routing information Traffic interception
ICMP Tunneling Hiding data in ICMP packets Firewall bypass, data exfiltration

Countermeasures

  • ICMP rate limiting: Apply rate limiting to ICMP traffic
  • Block unnecessary ICMP types: Block Redirect, Timestamp, etc.
  • Disable broadcast ping: Prevent Smurf attacks
  • Deep Packet Inspection (DPI): Detect ICMP tunneling
# Linux iptables ICMP rate limiting
iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s --limit-burst 4 -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

# Ignore ICMP Redirect settings
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects

5. Transport Layer Security (Layer 4)

The transport layer provides end-to-end communication through TCP and UDP protocols.

5.1 TCP SYN Flood

A classic DoS attack that exploits vulnerabilities in the TCP 3-way handshake.

TCP 3-Way Handshake Review

Normal TCP Connection Establishment:

Client                 Server
    |                    |
    |------- SYN ------->|  (Connection request)
    |                    |
    |<---- SYN-ACK ------|  (Connection accepted)
    |                    |
    |------- ACK ------->|  (Confirmation)
    |                    |
    |==== Connected =====|

SYN Flood Attack Principle

SYN Flood Attack:

Attacker                Server
    |                    |
    |------- SYN ------->|  (SYN sent with spoofed IP)
    |<---- SYN-ACK ------|  (Server waits for response - Half-Open connection)
    |------- SYN ------->|
    |<---- SYN-ACK ------|
    |------- SYN ------->|
    |<---- SYN-ACK ------|
    ...
    (Thousands of Half-Open connections created)

Result: Server's connection table fills up → Legitimate users cannot connect

Countermeasures

  • SYN Cookies: Manage connections with encrypted cookies without using SYN queue
  • SYN Proxy: Firewall/load balancer completes handshake before forwarding to server
  • Rate Limiting: Limit SYN packets per IP
  • Increase backlog queue: Increase allowed Half-Open connections
  • Reduce timeout: Shorten Half-Open connection hold time
# Enable SYN Cookies on Linux
echo 1 > /proc/sys/net/ipv4/tcp_syncookies

# Increase SYN backlog queue
echo 2048 > /proc/sys/net/ipv4/tcp_max_syn_backlog

# Reduce SYN-ACK retries
echo 2 > /proc/sys/net/ipv4/tcp_synack_retries

5.2 Port Scanning

Port scanning is a technique for identifying open ports and running services on a target system. It's primarily used during the reconnaissance phase.

Major Scan Types

Scan Type Method Characteristics
TCP Connect Scan Complete TCP connection establishment Accurate but logged
SYN Scan (Half-Open) Send SYN only, RST instead of ACK Fast and stealthy
FIN/NULL/Xmas Scan Using abnormal flag combinations Firewall bypass attempt
UDP Scan Sending UDP packets Slow but detects UDP services
ACK Scan Sending only ACK flag Firewall rule mapping 
# Various Nmap scan examples

# TCP Connect scan
nmap -sT 192.168.1.100

# SYN scan (requires admin privileges)
nmap -sS 192.168.1.100

# UDP scan
nmap -sU 192.168.1.100

# Service version detection
nmap -sV 192.168.1.100

# OS detection
nmap -O 192.168.1.100

# Full port scan
nmap -p- 192.168.1.100

Countermeasures

  • Close unnecessary ports: Disable unused services
  • Firewall rules: Allow only authorized IPs/ports
  • IDS/IPS: Detect and block scan patterns
  • Port knocking: Open ports with specific sequences
  • Honeypots: Lure attackers with fake services for analysis

5.3 Other Transport Layer Attacks

  • Session Hijacking: Taking over existing sessions by predicting TCP sequence numbers
  • TCP RST Attack: Forcing connection termination with forged RST packets
  • UDP Flood: Exhausting bandwidth with massive UDP packets

6. Application Layer Security (Layer 7)

The application layer directly interacts with users and runs various protocols and applications.

6.1 Major Application Layer Threats

  • SQL Injection: Manipulating database queries
  • XSS (Cross-Site Scripting): Inserting malicious scripts
  • CSRF (Cross-Site Request Forgery): Exploiting user privileges
  • DNS Attacks: DNS spoofing, DNS amplification, cache poisoning
  • HTTP Attacks: Slowloris, HTTP Flood
  • Email Attacks: Phishing, spam, malicious attachments

6.2 Countermeasures

  • WAF (Web Application Firewall): Block web application attacks
  • Input validation: Validate and escape all user input
  • DNSSEC: Verify DNS response integrity
  • Email security: Apply SPF, DKIM, DMARC
  • Enforce HTTPS: Encrypt all communications

7. Packet Analysis Basics with Wireshark

Wireshark is the most widely used open-source packet analysis tool. It's essential for network troubleshooting, security analysis, and protocol learning.

7.1 Wireshark Basic Interface

Wireshark Main Screen Structure:

+--------------------------------------------------+
| Menu Bar / Tool Bar                              |
+--------------------------------------------------+
| Filter Bar (Display Filter)                      |
+--------------------------------------------------+
|                                                  |
| Packet List Pane                                 |
| - Summary of all captured packets                |
|                                                  |
+--------------------------------------------------+
|                                                  |
| Packet Details Pane                              |
| - Layer-by-layer details of selected packet      |
|                                                  |
+--------------------------------------------------+
|                                                  |
| Packet Bytes Pane                                |
| - Raw data of selected packet (hexadecimal)      |
|                                                  |
+--------------------------------------------------+

7.2 Basic Display Filters

Filter Description
ip.addr == 192.168.1.100 Packets related to specific IP
tcp.port == 80 Specific TCP port
http HTTP protocol only
dns DNS traffic only
tcp.flags.syn == 1 Packets with SYN flag set
arp ARP packets only
icmp ICMP packets only
!(arp or dns or icmp) Exclude ARP, DNS, ICMP
tcp.analysis.flags TCP analysis flags (retransmissions, etc.)
frame contains "password" Packets containing specific string

7.3 Security Analysis Examples

ARP Spoofing Detection

# Filter ARP packets
Filter: arp

# Find suspicious patterns
- Different MAC addresses responding for the same IP
- Abnormally high number of ARP Replies
- Multiple Gratuitous ARP packets

# Useful filters
arp.duplicate-address-detected
arp.opcode == 2  # ARP Reply only

SYN Flood Detection

# Filter SYN packets
Filter: tcp.flags.syn == 1 && tcp.flags.ack == 0

# Analysis points
- Large volume of SYN packets to same destination
- Various source IPs (spoofing indication)
- SYN continues without SYN-ACK

# Check statistics
Statistics > Conversations > TCP tab
Statistics > Endpoints

Port Scan Detection

# Check connection attempts to various ports
Filter: ip.src == [suspicious IP] && tcp.flags.syn == 1

# Analysis points
- SYN sent to many ports in short time
- Sequential port number patterns
- Many RST responses (closed ports)

# Useful features
Statistics > Conversations
Statistics > Protocol Hierarchy

7.4 Practice Tips

  • Capture filter vs Display filter: Capture filters apply during saving, display filters apply during viewing
  • Follow TCP Stream: View entire conversation of a specific connection
  • Expert Information: View automatically analyzed warnings and errors
  • Protocol statistics: Various analysis available in Statistics menu
  • Save and share: Save in pcap/pcapng format to share analysis

Warning: Packet capture with Wireshark should only be performed on authorized networks. Unauthorized packet capture is illegal.

8. Layer-by-Layer Security Summary

Layer Major Threats Key Countermeasures
Physical (1) Wiretapping, equipment damage Physical access control, shielded cables
Data Link (2) ARP spoofing, MAC flooding DAI, port security, 802.1X
Network (3) IP spoofing, ICMP attacks Ingress filtering, uRPF, ICMP limiting
Transport (4) SYN Flood, port scanning SYN Cookies, IDS/IPS, firewalls
Application (7) SQL injection, XSS, phishing WAF, input validation, HTTPS

Conclusion and Next Episode Preview

In this article, we examined security threats and countermeasures for each layer of the OSI 7-layer model. Each layer has unique vulnerabilities, and comprehensive defense across all layers is necessary for effective security.

Attacks like ARP spoofing, IP spoofing, and TCP SYN Flood are threats that frequently occur in real-world scenarios, so it's important to understand the principles accurately and prepare appropriate countermeasures. Using packet analysis tools like Wireshark helps develop the ability to detect and analyze such attacks.

In the next episode, we'll cover types and configuration of firewalls, IDS/IPS systems, and Network Access Control (NAC). We'll discuss practical content about how actual network security devices work and how to configure them.

Recommended Practice: Set up a virtual environment (VirtualBox, VMware) to capture and analyze packets directly with Wireshark. You'll develop practical skills that can't be gained from theory alone.