AWS EC2 Ubuntu Server Security Hardening with Multi-Volume Architecture

Project Overview

Over nearly a decade (2016-2025), I have developed and implemented a comprehensive security hardening solution for AWS EC2 Ubuntu servers serving multiple production environments. This project addresses the critical need for secure, stable, and maintainable infrastructure through automated AMI creation processes that enforce strict security controls from the operating system level up.

The solution focuses on implementing a multi-volume encrypted architecture as the foundation for security and operational stability, complemented by system-level hardening, access controls, and monitoring capabilities. This approach enables rapid deployment of consistently secured servers across multiple client environments while maintaining strict separation of concerns at the filesystem level.

Business Challenge

Organizations running critical applications on AWS EC2 require infrastructure that balances security, performance, and operational efficiency. Several key challenges emerged across multiple client projects:

The solution needed to provide a repeatable, automated process for deploying hardened Ubuntu servers that enforce security best practices by design, not as an afterthought.

Technical Solution

Multi-Volume Encrypted Architecture

The cornerstone of this security approach is proper disk partitioning with encrypted, separated volumes for different filesystem purposes. This architecture provides both security and operational resilience by isolating different types of data and processes.

As detailed in my blog post The Importance of Proper Disk Partitioning in Linux, separating filesystems into dedicated volumes provides critical benefits:

Standard Volume Layout:

Volume	Purpose	Mount Options	Server Type
/	Root filesystem	defaults,discard,noatime,errors=remount-ro	All
/boot	Boot partition	defaults,noatime	All
/home	User directories	defaults,noatime,acl,user_xattr,nodev,nosuid	All
/usr	Applications	defaults,noatime,nodev,errors=remount-ro	All
/var	Variable data	defaults,noatime,nodev,nosuid	All
/tmp	Temporary files	tmpfs,defaults,nodev,noexec,nosuid,size=512m	All
swap	Memory management	sw	All
/var/www	Web data	defaults,noatime,acl,user_xattr,nodev,nosuid	Web servers
/var/lib/db	Database storage	defaults,noatime,nodev,nosuid	Database servers

Key Security Features:

This architecture is implemented through automated AWS EBS volume creation, attachment, partitioning, formatting, data synchronization, and /etc/fstab configuration during AMI build process.

System Hardening Components

Beyond disk architecture, the solution implements comprehensive system-level security controls:

SSH Security Hardening

• Disable password authentication (key-based only)
• Disable root login
• Configure AllowUsers whitelist for authorized accounts
• Custom SSH keep-alive settings for session management
• Restrict environment variable acceptance
• Key-based authentication with strong cryptographic keys: RSA 4096-bit initially, migrating to Ed25519 keys for enhanced security (stronger encryption with smaller key sizes, faster operations, and better resistance to side-channel attacks thanks to elliptic curve cryptography)

User and Access Management

• Rename default ubuntu user to customer-specific admin account
• Configure role-based sudo access with logging
• Set strong root password (though root login disabled via SSH)
• Generate customer-specific SSH keys for automated tasks

Package Management and System Utilities

Installation of essential security and monitoring tools:

• fail2ban for intrusion prevention
• acl for advanced file permissions
• System monitoring tools (htop, sysstat, goaccess)
• Network diagnostic utilities (traceroute, net-tools)
• Security utilities (policycoreutils)

Shell and Environment Configuration

• Install and configure zsh with oh-my-zsh for admin users
• Enhanced shell history with timestamps (HISTTIMEFORMAT)
• Increased history size for audit trail
• Custom vim configuration as default editor
• Command-line efficiency improvements

Service-Specific Hardening

Tailored security configurations based on server role:

Automated AMI Build Process

All security configurations are codified in automated Bash scripts that:

1. Stop the source EC2 instance
2. Create and attach encrypted EBS volumes
3. Partition, format, and mount new volumes
4. Synchronize data from root volume to appropriate partitions
5. Generate optimized /etc/fstab configuration
6. Apply all security hardening steps
7. Create snapshots and build new AMI
8. Verify and tag the resulting AMI

This approach ensures:

• Consistency: Every server built from the AMI has identical security posture
• Repeatability: New environments can be deployed rapidly with confidence
• Version control: AMIs are versioned and can be rolled back if needed
• Audit trail: AWS tracks all AMI creation and usage

Configuration Management Integration

The solution integrates with AWS CodeCommit for version-controlled configuration management:

• Apache virtual host configurations stored in Git
• PHP pool configurations under version control
• Custom scripts and configurations tracked and deployed automatically
• Infrastructure-as-code approach for server configurations

Additional Security Layers

Beyond the core hardening measures, the solution incorporates additional security controls that provide defense-in-depth protection:

Automatic Security Updates

• Unattended security patches: Configure automatic installation of critical security updates
• Staged rollout strategy: Test updates on non-production instances before applying to production
• Reboot management: Controlled instance restarts when kernel updates require them
• Update monitoring: Track installed patches and maintain update history for compliance

Kernel-Level Hardening

System kernel parameters (sysctl) hardened for enhanced security:

• Network security: IP forwarding controls, SYN flood protection, ICMP redirect prevention
• Memory protection: Address space layout randomization (ASLR), kernel pointer protection
• Core dump restrictions: Prevent sensitive data exposure through core dumps
• Process isolation: Enhanced ptrace restrictions to prevent process debugging attacks

Network Security and Firewall

Multi-layered network security approach (see my comprehensive AWS Network Security Architecture project for full implementation details):

• Host-based firewall: iptables/nftables rules for granular port and protocol control
• AWS Security Groups: Cloud-level firewall integrated with instance-level controls
• VPC network segmentation: Isolation of different server roles in separate subnets
• Egress filtering: Restrict outbound connections to prevent data exfiltration

File Integrity Monitoring

Detection of unauthorized filesystem modifications:

• Baseline creation: Establish cryptographic hashes of critical system files
• Scheduled verification: Regular scans to detect unauthorized changes
• Alert generation: Immediate notification of integrity violations
• Compliance support: Maintain audit trails for security compliance requirements

Centralized Logging and Monitoring

Comprehensive observability and security event tracking:

• CloudWatch integration: System and application logs forwarded to AWS CloudWatch
• Log retention policies: Configurable retention periods for compliance requirements
• Real-time alerting: Automated notifications for critical security events
• SIEM integration: Support for forwarding logs to Security Information and Event Management systems
• Audit log protection: Immutable log storage prevents tampering

Rootkit and Malware Detection

Proactive threat detection capabilities:

• Rootkit scanners: Regular scans for known rootkits and backdoors
• Binary integrity verification: Detect trojaned system binaries
• Malware signature updates: Automatic updates of threat detection databases
• Scheduled scanning: Automated security scans during low-traffic periods

Backup and Disaster Recovery

Automated backup strategy for data protection:

• EBS snapshot automation: Scheduled snapshots of all data volumes
• Cross-region replication: Geographic redundancy for critical data
• Backup verification: Automated tests to ensure backup integrity
• Point-in-time recovery: Ability to restore to any snapshot timestamp
• Retention management: Automated cleanup of old snapshots per policy

AWS Systems Manager Integration

Centralized management and security:

• Session Manager: Secure shell access without SSH key distribution
• Patch Manager: Automated patch baseline enforcement
• Parameter Store: Secure storage for configuration values and secrets
• Systems Manager Agent: Automated inventory and compliance reporting
• Run Command: Remote execution capabilities for operational tasks

Compliance and Audit Support

Built-in compliance framework support:

• CIS Benchmark alignment: Configuration follows Center for Internet Security guidelines
• PCI-DSS considerations: Payment Card Industry compliance support
• HIPAA security controls: Healthcare data protection measures where applicable
• SOC 2 evidence collection: Audit trail generation for SOC 2 compliance
• Automated compliance scanning: Regular assessment against security baselines

Implementation Results

This security hardening solution has been successfully deployed across multiple production environments since 2019, serving various industries including finance, e-commerce, healthcare, and SaaS platforms.

Key Outcomes:

• Zero filesystem-related outages due to partition isolation
• Consistent security posture across all deployed environments
• Rapid environment provisioning: New secure servers deployed in minutes, not hours
• Simplified compliance auditing: Security controls built into base infrastructure
• Reduced attack surface: Multi-layered security prevents common exploitation vectors
• Enhanced disaster recovery: Volume-level backups enable granular restoration

Security Improvements:

• Prevented lateral movement in potential breach scenarios through filesystem isolation
• Eliminated risk of log-based denial of service attacks
• Protected against temporary file exploitation with noexec mounts
• Enforced least-privilege access through role-based user management
• Created comprehensive audit trails for access and configuration changes

Technical Expertise Demonstrated

This project showcases expertise in:

• AWS Infrastructure: EC2, EBS, AMI, snapshots, encryption at rest
• Linux System Administration: Advanced filesystem management, security hardening, user/permission management
• Security Engineering: Defense in depth, principle of least privilege, secure defaults
• Automation and DevOps: Infrastructure as code, repeatable deployments, configuration management
• Server Role Specialization: Tailored hardening for bastion hosts, web servers, database servers
• Compliance and Auditing: Security logging, access controls, session recording

Relevance for Future Projects

Organizations seeking robust, secure, and maintainable AWS infrastructure will benefit from:

• Proven multi-volume architecture approach for security and stability
• Automated, repeatable deployment processes
• Comprehensive security hardening that goes beyond basic configurations
• Integration capabilities with existing AWS security services
• Scalable approach suitable for both small deployments and large enterprise environments

This solution demonstrates the value of security by design - building security into infrastructure from the ground up rather than retrofitting it later. The multi-volume architecture provides a solid foundation that makes systems inherently more secure, stable, and manageable.

---