Running containerized infrastructure on low-resource environments is always a balancing act. Recently, I ran into an interesting stability issue with a WordPress deployment hosted on a 1 CPU / 1 GB RAM DigitalOcean droplet.
The symptom was straightforward but annoying: the MySQL database container (wp_db) was silently stopping every week or so, bringing the site down with it. Here is a breakdown of how I diagnosed the issue, what was happening under the hood, and how to stabilize MySQL 8 in a heavily memory-constrained environment.
The Symptoms: Long Semaphore Waits
Taking a look at the MySQL container logs (docker logs wp_db), the database wasn’t shutting down gracefully. Instead, every time it booted up, it ran through an XA crash recovery.
Digging deeper into the logs, a pattern emerged. Right before the crashes, InnoDB was throwing these warnings:
[Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 140580524979776 has waited at log0meb.cc line 1781 for 1449 seconds the semaphore:
A “semaphore wait” means a MySQL thread is stuck waiting for a locked resource (like memory or disk I/O). When a thread hangs for too long, InnoDB intentionally crashes the database engine as a built-in safety mechanism to prevent data corruption. But why were the threads hanging in the first place?
The Diagnosis: The OOM Killer Strikes
Since the host only had 1 GB of RAM, memory exhaustion was the prime suspect. To confirm, I checked the host’s kernel logs for Out-Of-Memory (OOM) events:
dmesg -T | grep -i oom
The output was the smoking gun:
Out of memory: Killed process 1864704 (mysqld) total-vm:1298212kB, anon-rss:390228kB...
The Linux kernel’s OOM killer was stepping in and terminating mysqld to keep the host OS alive. Running free -h revealed the final piece of the puzzle: Swap: 0B.
MySQL 8.0 has a significantly higher default memory footprint compared to older versions. When the system ran out of physical RAM and had absolutely zero swap space to fall back on, the kernel panicked and killed the process. This abrupt termination caused the InnoDB semaphore hangs and the subsequent crash recoveries.
The Fix: Adding Swap and Throttling MySQL
To stabilize the environment, we need to attack the problem from two angles: give the host a memory buffer, and restrict MySQL’s default memory footprint.
Step 1: Create a Swap File
Adding a 1 GB swap file gives the OS a safety net when physical RAM peaks.
sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
To make this persistent across reboots, add it to /etc/fstab:
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
Step 2: Create a Memory-Optimized my.cnf
MySQL 8 assumes it has plenty of resources. We need to explicitly throttle it by creating a custom low-mem.cnf file:
[mysqld]
# Disable Performance Schema (Saves ~150MB+ of RAM)
performance_schema = 0
# Restrict the buffer pool (The core cache)
innodb_buffer_pool_size = 64M
# Limit maximum connections to prevent thread-thrashing
max_connections = 40
# Lower per-thread buffers
read_buffer_size = 256K
read_rnd_buffer_size = 512K
sort_buffer_size = 512K
join_buffer_size = 512K
# Restrict temporary tables
tmp_table_size = 16M
max_heap_table_size = 16M
Step 3: Mount the Config in docker-compose.yml
Finally, map this new configuration file directly into the MySQL container using a read-only bind mount.
services:
db:
....
volumes:
.....
# Mount the custom low-memory config
- ./low-mem.cnf:/etc/mysql/conf.d/low-mem.cnf:ro
networks:
......
After updating the compose file, a quick docker compose down and docker compose up -d applies the new limits.
Moving Forward
These optimizations should prevent the OS from triggering the OOM killer and keep the container stable. However, 1 GB of RAM is still a tight squeeze for MySQL 8.0.
If memory constraints continue to be a bottleneck despite these tuning efforts, the next logical step will be swapping the mysql:8.0 image for mariadb:10.6. MariaDB serves as an excellent drop-in replacement with a notably lower baseline memory utilization, making it highly suited for micro-environments.
