Home Server Complete Guide Part 7: Security Settings and Firewall

Essential Security Measures to Protect Your Server

2026-01-22 Technology

Introduction: Why Home Server Security Matters

The moment you expose your home server to the internet, bots and hackers from around the world start scanning it. This is not an exaggeration. When you put a new server online, you can see SSH brute force attacks starting within minutes in the logs. If you've completed network and DDNS setup from the previous guide, it's now time to learn how to protect your server.

Security may seem difficult and complex, but following basic principles will block most attacks. In this guide, we'll cover essential security configurations that every home server operator should know.

1. UFW Firewall Configuration

1.1 What is a Firewall?

A firewall is a security system that controls network traffic going in and out of your server. Simply put, it determines "who can enter through which door." On Ubuntu, UFW (Uncomplicated Firewall) provides an easier way to manage firewalls compared to iptables.

1.2 UFW Installation and Basic Setup

UFW comes pre-installed on most Ubuntu versions. Check if it's installed and install if needed:

# Check and install UFW
sudo apt update
sudo apt install ufw

# Check UFW status
sudo ufw status

It will initially be disabled. Before enabling, you must allow SSH access. Otherwise, you'll be locked out of the server the moment the firewall activates.

# Set default policies: deny all incoming, allow all outgoing
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH (important!)
sudo ufw allow ssh
# Or specify by port number
sudo ufw allow 22/tcp

# Allow web server ports
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Enable UFW
sudo ufw enable

1.3 Allowing Ports by Service

Open necessary ports based on the services you're running:

# Allow specific port
sudo ufw allow 8080/tcp    # Alternative web port
sudo ufw allow 32400/tcp   # Plex media server

# Allow port range
sudo ufw allow 6000:6010/tcp

# Allow access only from specific IP
sudo ufw allow from 192.168.1.0/24 to any port 22

# Block specific IP
sudo ufw deny from 123.45.67.89

# Check rules
sudo ufw status numbered

# Delete specific rule (by number)
sudo ufw delete 3

Warning: If you're managing the server remotely, be extra careful not to block the SSH port. If you accidentally block it, you'll need physical access to the server.

2. Defending Against Brute Force Attacks with fail2ban

2.1 What is fail2ban?

fail2ban monitors log files and automatically blocks IPs that show abnormal access patterns (such as repeated login failures). It's very effective at stopping SSH brute force attacks.

2.2 Installing and Configuring fail2ban

# Install fail2ban
sudo apt install fail2ban

# Start service and enable on boot
sudo systemctl start fail2ban
sudo systemctl enable fail2ban

fail2ban configuration is located in /etc/fail2ban/. It's recommended to create a local file instead of modifying the default configuration directly:

# Create local configuration file
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local

2.3 Detailed jail.local Configuration

Let's look at the key configuration options:

[DEFAULT]
# Ban duration (in seconds, -1 for permanent)
bantime = 3600

# Time window to monitor login attempts
findtime = 600

# Allowed number of failures
maxretry = 5

# IPs to exclude from banning (add your own IP)
ignoreip = 127.0.0.1/8 ::1 192.168.1.0/24

# Email notification on ban (optional)
# action = %(action_mwl)s
# destemail = your@email.com
# sender = fail2ban@yourdomain.com

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 86400

# Nginx-related jails
[nginx-http-auth]
enabled = true
port = http,https
filter = nginx-http-auth
logpath = /var/log/nginx/error.log
maxretry = 5

[nginx-botsearch]
enabled = true
port = http,https
filter = nginx-botsearch
logpath = /var/log/nginx/access.log
maxretry = 2

Restart fail2ban after changing settings:

# Apply configuration
sudo systemctl restart fail2ban

# Check current ban status
sudo fail2ban-client status
sudo fail2ban-client status sshd

# Manually unban specific IP
sudo fail2ban-client set sshd unbanip 123.45.67.89

3. Hardening SSH Security

3.1 Changing SSH Port

Changing the default SSH port (22) helps avoid a significant portion of automated bot attacks. It's not a perfect security measure, but you'll notice a significant reduction in attack logs.

# Edit SSH configuration file
sudo nano /etc/ssh/sshd_config

# Find and modify the following line (uncomment and change port)
Port 2222

Make sure to allow the new port in UFW before changing:

# Allow new SSH port
sudo ufw allow 2222/tcp

# Restart SSH service
sudo systemctl restart sshd

# Connect using new port
ssh -p 2222 username@server_ip

3.2 Allowing Only SSH Key Authentication

SSH key authentication is much more secure than password authentication. We strongly recommend setting up key authentication and disabling password login.

# Generate SSH key on your client (your PC)
ssh-keygen -t ed25519 -C "your_email@example.com"

# Copy public key to server
ssh-copy-id -p 2222 username@server_ip

# Or copy manually
cat ~/.ssh/id_ed25519.pub | ssh -p 2222 username@server_ip "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

After confirming key authentication works, disable password authentication:

# Edit SSH configuration file
sudo nano /etc/ssh/sshd_config

# Find and modify these settings
PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

# Restart SSH
sudo systemctl restart sshd

Important: Before disabling password authentication, make sure you can log in with your SSH key. Otherwise, you could be completely locked out of the server.

3.3 Additional SSH Security Settings

# Additional settings in /etc/ssh/sshd_config
# Allow only specific users for SSH access
AllowUsers username1 username2

# Limit connection grace time
LoginGraceTime 30

# Maximum authentication attempts
MaxAuthTries 3

# Limit unauthenticated connections
MaxStartups 10:30:60

# Disable X11 forwarding (if not needed)
X11Forwarding no

4. SSL/TLS Certificates (Let's Encrypt)

4.1 Why You Need HTTPS

HTTP communication is unencrypted, allowing data to be intercepted in transit. HTTPS uses SSL/TLS certificates to encrypt communication and verify server identity. With Let's Encrypt, you can obtain trusted certificates for free.

4.2 Installing Certbot and Obtaining Certificates

# Install Certbot
sudo apt install certbot python3-certbot-nginx

# For Nginx (automatic configuration)
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com

# Certificate only (manual configuration)
sudo certbot certonly --standalone -d yourdomain.com

# Verify certificate issuance
sudo certbot certificates

4.3 Automatic Certificate Renewal

Let's Encrypt certificates must be renewed every 90 days. Certbot sets up an automatic renewal timer:

# Check automatic renewal timer
sudo systemctl status certbot.timer

# Test manual renewal
sudo certbot renew --dry-run

# Set up cron for automatic renewal (optional)
sudo crontab -e
# Add the following line (attempt renewal daily at 3 AM)
0 3 * * * /usr/bin/certbot renew --quiet

5. Reverse Proxy and HTTPS

5.1 Nginx Reverse Proxy Configuration

Using a reverse proxy allows you to route multiple services through one server using domains or paths. It also makes SSL certificate management convenient by handling it in one place.

# /etc/nginx/sites-available/reverse-proxy
server {
    listen 80;
    server_name yourdomain.com;

    # Redirect HTTP to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name yourdomain.com;

    # SSL certificate configuration
    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

    # SSL security settings
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
    ssl_prefer_server_ciphers off;

    # HSTS header
    add_header Strict-Transport-Security "max-age=63072000" always;

    # Proxy to main application
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }

    # Proxy to API server
    location /api/ {
        proxy_pass http://localhost:8080/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

# Enable configuration
sudo ln -s /etc/nginx/sites-available/reverse-proxy /etc/nginx/sites-enabled/

# Test and apply configuration
sudo nginx -t
sudo systemctl reload nginx

6. Automating Security Updates

6.1 Configuring unattended-upgrades

Missing security updates can expose you to known vulnerabilities. Ubuntu's unattended-upgrades can automatically apply security updates:

# Install packages
sudo apt install unattended-upgrades apt-listchanges

# Configure automatic updates
sudo dpkg-reconfigure -plow unattended-upgrades

6.2 Detailed Configuration

# Edit configuration file
sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

Key configuration options:

// Package sources for automatic updates
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}";
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";
};

// Automatically remove unused kernel packages
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";

// Automatic reboot (when necessary)
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:00";

// Email notifications
Unattended-Upgrade::Mail "your@email.com";
Unattended-Upgrade::MailReport "on-change";

7. Introduction to Intrusion Detection

7.1 Log Monitoring

Regularly checking server logs helps you detect anomalies early:

# Check authentication logs (SSH login attempts)
sudo tail -f /var/log/auth.log

# Check failed login attempts
sudo grep "Failed password" /var/log/auth.log | tail -20

# Check successful logins
sudo grep "Accepted" /var/log/auth.log | tail -20

# Check system logs
sudo journalctl -xe

# Recent login history
last
lastb  # Failed logins

7.2 File Integrity Checking with AIDE

AIDE (Advanced Intrusion Detection Environment) detects changes to the file system:

# Install AIDE
sudo apt install aide

# Create initial database
sudo aideinit

# Apply database
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Run integrity check
sudo aide --check

# Set up regular checks via cron
sudo crontab -e
# Check daily at 4 AM
0 4 * * * /usr/bin/aide --check | mail -s "AIDE Report" your@email.com

7.3 Rootkit Scanning with rkhunter

# Install rkhunter
sudo apt install rkhunter

# Update database
sudo rkhunter --update

# Update properties
sudo rkhunter --propupd

# Run rootkit scan
sudo rkhunter --check

# Set up automatic scanning (cron)
sudo crontab -e
0 5 * * * /usr/bin/rkhunter --check --skip-keypress 2>&1 | mail -s "rkhunter Report" your@email.com

8. Security Checklist

Here's a summary checklist for home server security:

Item	Status	Priority
UFW Firewall Enabled	Required	High
SSH Key Authentication Set Up	Required	High
SSH Password Authentication Disabled	Recommended	High
fail2ban Installed and Configured	Recommended	High
SSH Port Changed	Optional	Medium
Root Login Disabled	Recommended	High
SSL/TLS Certificate Applied	Required (web services)	High
Automatic Security Updates	Recommended	High
Regular Log Checking	Recommended	Medium

Conclusion

Security isn't something you configure once and forget - it requires ongoing management. The basic security settings covered in this guide will protect your server from most automated attacks.

In the next part, we'll cover backup strategies and monitoring, which are essential for server operations. No matter how thorough your security is, unexpected situations can occur, so preparing with backups and monitoring is wise.

If you have questions or issues, please leave a comment. We'll work through them together.

Back to List