High Server Load: How to Find the Cause in 15 Minutes

In short. Start with uptime and nproc — load average is compared against core count, not against 1. Then vmstat 1 separates three different kinds of load: CPU queue (column r), disk wait (b and wa), and swapping (si/so). Next come free -h, df -h and df -i, iostat -x 1, ss -s. You name the culprit with ps aux --sort=-%cpu and the nginx, PHP-FPM and database logs.

"The server is slow" is a symptom, not a diagnosis. The same external picture — slow responses, or 502/503 errors — can be caused by CPU starvation, disk wait, swapping, a full filesystem, a connection spike, one unindexed SQL query, or a wave of scrapers. Each has a different fix, and adding cores helps in exactly one of those cases. Below is a command sequence that narrows the problem down to a single layer in 15 minutes, plus what to do next.

The first 60 seconds: load average, top and vmstat

The goal of minute one is not to find the guilty process but to identify which resource is exhausted. Three commands give you 80% of the answer.

uptime
nproc
vmstat 1 5
free -h
df -h; df -i

Reading load average, and why "load 4" on 4 cores is not always bad

The uptime output ends with three numbers: average load over 1, 5 and 15 minutes.

$ uptime
 14:22:31 up 82 days,  3:11,  2 users,  load average: 4.12, 3.87, 2.05
$ nproc
4

On Linux, load average counts tasks that are running, waiting for CPU, or sitting in uninterruptible sleep (state D — usually waiting on disk I/O). That is the key difference from classic UNIX: on Linux, disk wait is folded into load, so a load of 30 with an idle CPU is a perfectly normal picture for a box that is bottlenecked on storage.

Always normalise load by core count. Load 4.12 on four cores means roughly 100% utilisation: the machine is at its limit but the queue is not exploding yet. The same 4.12 on 16 cores is a quarter of capacity and nothing to worry about. The same 4.12 on a single core means each task waits in queue about three times longer than it runs.

Comparing the three numbers matters more than their absolute value. A series of 4.12 / 3.87 / 2.05 means load is rising — it started recently and is getting worse. The reverse, 2.05 / 3.87 / 4.12, means the peak has passed and you may be diagnosing the aftermath rather than the cause. A flat series is a steady state, and it should be compared against history, not against intuition.

Load average never answers "why". It only answers "how many tasks are waiting". Until you separate CPU wait from disk wait, any action you take is guesswork.

Which columns to read in top and htop

Not every column in top is useful. Four of them are:

• %CPU — share of a single core. 100% means one fully busy core, not the whole box. A multithreaded process will happily show 380% on a four-core machine, and that is fine.
• RES — resident memory actually held in RAM. This, not VIRT, is what you inspect when you suspect a leak. VIRT includes mapped files and reserved ranges and almost always looks alarmingly large.
• S — process state. R running or runnable, S sleeping, D uninterruptible sleep (waiting on disk or network inside the kernel), Z zombie. Several processes in D at once is nearly always a storage problem, not a CPU problem.
• The %Cpu(s) line — the breakdown: us (user code), sy (kernel), wa (I/O wait), st (steal — cycles the hypervisor took away from your VM).

Press 1 in top to expand the CPU line per core. It often turns out that one core out of eight is pinned, which means a single-threaded process is the bottleneck and upgrading to 16 cores will change nothing. htop is nicer to read but exposes essentially the same data.

Sustained st above a few percent on a VPS means the hypervisor is not giving you the cycles you pay for. That is host oversubscription or a plan limit, and it cannot be fixed from inside the virtual machine.

vmstat 1: columns r, b, si, so and wa

One command that separates three kinds of load. Ignore the first line — it is an average since boot.

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 6  0      0 152344  84120 2210436    0    0     0    28  912 1840 94  4  2  0  0
 5  0      0 151980  84120 2210436    0    0     0     0  880 1792 96  3  1  0  0

• r — processes in the run queue. Consistently above core count means you are CPU bound.
• b — processes in uninterruptible sleep waiting on I/O. Persistently above zero means you are disk bound.
• si / so — kilobytes per second swapped in and out. Non-zero values right now mean active swapping, the most painful kind of slowness.
• wa — share of CPU time spent waiting for I/O. High wa with low us means the processor is idle, waiting for storage.
• cs — context switches. A sharp rise at constant request volume points to too many workers or threads.

Three scenarios read instantly:

vmstat pattern	What it means	Where to dig
r above core count, wa near 0, us high	CPU starvation	Which process: ps aux --sort=-%cpu, PHP, builds, antivirus, cron
b above 0, wa high, us low	I/O starvation	iostat -x 1, iotop, processes in state D
si and so non-zero, free tiny	Swapping	RAM shortage: free -h, then ps aux --sort=-%mem
sy high, cs huge, us low	Kernel overhead	Too many workers, interrupt storm, network flood
st consistently above zero	Steal on a VM	Host oversubscription or plan cap — a question for the provider

Memory: free -h, page cache, swap and the OOM killer

The most common false alarm is "almost no free memory left". On Linux that is normal. The kernel uses all otherwise idle RAM as page cache for files on disk, and hands it back to applications the instant they need it.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            7.7Gi       3.1Gi       210Mi       180Mi       4.4Gi       4.1Gi
Swap:           2.0Gi       512Mi       1.5Gi

Read available, not free. Available is the kernel's own estimate of how much memory a new process can get without swapping, counting the reclaimable part of the cache. Here free = 210 MiB looks frightening, while available = 4.1 GiB says memory is fine.

The real warning sign is available dropping to a few percent of total while vmstat shows non-zero si/so. That is swapping: the kernel pushes application pages to disk and reads them back, every memory access turns into a disk operation, and performance falls by orders of magnitude.

Swap and swappiness

The vm.swappiness knob controls how eagerly the kernel evicts anonymous pages instead of dropping page cache. The default on most distributions is 60.

cat /proc/sys/vm/swappiness
sudo sysctl -w vm.swappiness=10
# make it permanent:
echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf

# which processes are actually swapped out
for f in /proc/*/status; do
  awk '/^Name:|^VmSwap:/{printf "%s ", $2} END{print ""}' "$f"
done | sort -k2 -h -r | head -15

Lowering swappiness to 10 is reasonable for database and web servers: swap stays as insurance against OOM but stops being used preventively. Disabling swap entirely is a bad idea — without it, the kernel calls the OOM killer as soon as RAM runs out, so instead of degradation you get a dead process.

OOM killer: finding the evidence

If a process "just disappeared" and the application log says nothing, the OOM killer almost certainly fired.

sudo dmesg -T | grep -i -E 'out of memory|killed process|oom-kill'
sudo journalctl -k --since "2 hours ago" | grep -i oom
sudo journalctl -u php-fpm --since today | grep -i -E 'exited|killed'

The kernel line names the victim and reports its total-vm/rss at time of death. That is hard evidence: you know both what died and how big it was. The remaining question is why it grew — a leak, an oversized PHP memory_limit multiplied by too many workers, or a query pulling a million rows into memory.

A classic web-server failure: pm.max_children in PHP-FPM is set "with headroom" while memory_limit is 512 MB. During a traffic spike the workers together exceed RAM, swapping starts, then the OOM killer arrives and kills the database as the fattest process. Compute max_children as available memory divided by the real average worker footprint.

Disk: free space, inodes and deleted-but-open files

A full filesystem produces wildly varied symptoms: 502/503 from the web server, a database that refuses to start, a blank PHP page, sessions or logs that cannot be written. Checking takes five seconds.

df -h
df -i
df -h /var /tmp /var/lib/mysql

df -h vs df -i: "No space left on device" with a half-empty disk

ext-family filesystems store file metadata in inodes, and their count is fixed at format time. If an application creates millions of tiny files — cache entries, sessions, a mail queue, temp files — inodes run out long before bytes do. Then df -h reports 40% used while every write fails with "No space left on device".

$ df -i /
Filesystem       Inodes   IUsed IFree IUse% Mounted on
/dev/vda1       3276800 3276794     6  100% /

Find the directory holding millions of files:

# size of every top-level subdirectory
sudo du -xh --max-depth=1 / | sort -h | tail -20
sudo find /var -xdev -type d -exec sh -c 'echo "$(ls -U "$1" | wc -l) $1"' _ {} \; 2>/dev/null | sort -rn | head

Usual suspects: PHP session directories, mail spools, CMS cache trees, thumbnail directories, and queue backlogs that nobody ever pruned.

Finding what ate the space

# top-down, staying inside one filesystem
sudo du -xh --max-depth=1 / | sort -h
sudo du -xh --max-depth=1 /var | sort -h

# interactive, much faster to navigate
sudo ncdu -x /

# systemd journals love to grow
journalctl --disk-usage
sudo journalctl --vacuum-size=200M

The -x flag is mandatory: without it du wanders into /proc, network mounts and container volumes and returns meaningless numbers.

Deleted but still open files

A situation that stumps people: du accounts for 20 GB while df reports 90 GB used. Someone deleted a large file, but a process still holds the descriptor open — the space is only released when that process restarts.

sudo lsof +L1
# or only the large ones:
sudo lsof -nP +L1 2>/dev/null | awk '$7 > 100000000'

Most often it is a rotated but never reopened log: someone deleted logs by hand instead of using logrotate, and nginx or the application keeps writing into the deleted inode. The fix is nginx -s reopen, a systemctl reload, or a restart — plus a proper logrotate rule so it does not happen again. See log management practices for the details.

Keep 15–20% free on the database volume and on root. ext4 reserves roughly 5% of blocks for root, so the system still boots after ordinary processes can no longer write. That reserve is your rescue margin, not a working condition.

Disk I/O: iostat -x 1, %util and await

If vmstat showed a non-zero b column and high wa, move on to iostat (part of the sysstat package).

iostat -x 1
# a single device only
iostat -xd 1 vda

The columns that matter:

Column	What it shows	When to worry
r/s, w/s	Read and write operations per second (IOPS)	Flat-lining against a plan or device limit
rkB/s, wkB/s	Throughput	A constant ceiling at low IOPS means a bandwidth cap
r_await, w_await	Average service time per operation, ms	Single-digit ms is fine on SSD/NVMe; tens are already bad
aqu-sz	Average queue depth	Consistently above 1–2 means the device cannot keep up
%util	Share of time the device was serving requests	100% saturates an HDD; on NVMe the number is misleading

On NVMe and network-attached storage, %util is nearly useless: the device serves dozens of queues in parallel, and 100% does not mean saturation. Trust await and aqu-sz instead.

Who is actually doing the I/O:

sudo iotop -oPa            # active processes only, accumulated
pidstat -d 1               # same data, non-interactive
ps -eo state,pid,ppid,comm,wchan | awk '$1 ~ /^D/'

Common causes of an I/O spike on a web server: a nightly backup or replication, a full antivirus scan, a search index rebuild, an unindexed query doing a full table scan, log rotation and compression, and — on oversubscribed VPS plans — noisy neighbours on the host.

Network and connections: ss -s and a TIME-WAIT spike

Sometimes the load is not in resources but in connection count. The symptom is slow responses or errors while CPU and disk sit idle.

ss -s
ss -tan state time-wait | wc -l
ss -tan state established | wc -l
ss -ltn                       # listeners and their queues
ss -tan state syn-recv | wc -l

In ss -ltn, read Recv-Q and Send-Q for listening sockets: Recv-Q is the current number of accepted but unprocessed connections, Send-Q is the configured backlog. When Recv-Q pins against Send-Q, the application cannot accept fast enough and some clients get timeouts or resets.

Tens of thousands of sockets in TIME-WAIT are not a disaster by themselves — it is the normal post-close state and it clears in about a minute. The trouble starts when their number approaches the local port range and outbound connections (to the database, to an API, to an upstream) stop being established.

cat /proc/sys/net/ipv4/ip_local_port_range
# conntrack table limit, if netfilter is in play
cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count
sudo dmesg -T | grep -i 'nf_conntrack: table full'

The right cure for a TIME-WAIT spike is not blind sysctl tuning but enabling keep-alive between nginx and the upstream so connections get reused. The practical recipe is in nginx performance tuning. If the connection queue overflows under a traffic wave, see rate limiting strategies.

Naming the culprit: processes, nginx, PHP-FPM, MySQL

The resource is identified — now for the name.

ps aux --sort=-%cpu | head -15
ps aux --sort=-%mem | head -15
pidstat -u -p ALL 1 5 | sort -k8 -rn | head
systemd-cgtop                      # load per unit and container

nginx: finding slow requests

By default nginx does not log response time. Add it — without it there is nothing to search.

log_format timed '$remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent" '
                 'uct=$upstream_connect_time uht=$upstream_header_time '
                 'urt=$upstream_response_time rt=$request_time';

access_log /var/log/nginx/access.log timed;

After reloading the config (nginx -t && nginx -s reload), slow URLs are one line away:

# 20 slowest requests
awk -F'rt=' '{print $2, $0}' /var/log/nginx/access.log | sort -rn | head -20

# current requests per minute
awk '{print $4}' /var/log/nginx/access.log | cut -d: -f2,3 | uniq -c | tail -10

PHP-FPM: slowlog and worker count

; in the pool file, e.g. /etc/php-fpm.d/www.conf
slowlog = /var/log/php-fpm/www-slow.log
request_slowlog_timeout = 5s
request_terminate_timeout = 60s

pm = dynamic
pm.max_children = 20
pm.start_servers = 5
pm.min_spare_servers = 3
pm.max_spare_servers = 8
pm.max_requests = 500

The slowlog writes a full PHP stack trace at the moment the timeout is exceeded — an instant answer to "which function are we stuck in". To check pool health:

sudo systemctl status php-fpm
# with pm.status_path enabled
curl -s http://127.0.0.1/status?full | head -40
grep -c 'pool www' /var/log/php-fpm/www-slow.log

If the status page constantly shows a non-zero listen queue, you are short of workers. But before raising pm.max_children, do the arithmetic: workers times real per-worker memory must fit in RAM with headroom for the database and cache. Otherwise you simply trade a slow response for an OOM kill. How this looks from outside is covered in the 502 Bad Gateway and 503 Service Unavailable guides.

Database: PROCESSLIST and slow queries

-- what is running right now
SHOW FULL PROCESSLIST;

SELECT id, user, host, db, command, time, state, LEFT(info, 120) AS query
FROM information_schema.processlist
WHERE command <> 'Sleep'
ORDER BY time DESC
LIMIT 20;

-- turn on the slow query log
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;
SET GLOBAL log_queries_not_using_indexes = 'ON';

# aggregate the slow query log
sudo mysqldumpslow -s t -t 20 /var/log/mysql/slow.log

# inspect the plan of a suspicious query
mysql -e "EXPLAIN SELECT ...\G"

A single unindexed query against a multi-million-row table can take the whole server down: it reads the table from disk, evicts the page cache, drives up wa, and every other query starts waiting. This is the most common cause of "sudden" load on sites that ran fine for years — the table simply grew to the size where a full scan stopped being cheap.

Bots and scrapers or real traffic: reading access.log

Before optimising code, check whose traffic you are paying for. Parsing the log takes a minute.

# top IPs by request count
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# top User-Agents
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# top URLs
awk '{print $7}' /var/log/nginx/access.log | cut -d? -f1 | sort | uniq -c | sort -rn | head -20

# what one specific IP is doing
grep '^203.0.113.45 ' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

Signals that separate a scraper from humans:

• One IP or subnet accounts for a disproportionate share of requests — dozens per second with no pauses.
• Requests walk a catalogue strictly in order, or enumerate every combination of filter parameters. Combinatorial explosion on faceted-filter pages kills a server faster than most floods.
• No requests for static assets. A browser pulls CSS, fonts and images; a scraper takes only HTML.
• Referer is missing or forged; User-Agent is empty, identical across thousands of requests, or names an HTTP library.
• A self-declared Googlebot arrives from an address that reverse DNS does not confirm.

# verify that a crawler is genuine
host 66.249.66.1
# the answer must resolve into googlebot.com or google.com,
# then confirm the name forward:
host crawl-66-249-66-1.googlebot.com

What to do about it. Useful crawlers are not blocked — their crawl depth is limited: disallow parametric filter and sort URLs in robots.txt, serve correct canonical links, remove infinite calendars and endless pagination. Abusive clients get a rate limit at the nginx level:

limit_req_zone $binary_remote_addr zone=perip:10m rate=5r/s;
limit_conn_zone $binary_remote_addr zone=connperip:10m;

server {
    location / {
        limit_req  zone=perip burst=20 nodelay;
        limit_conn connperip 20;
        limit_req_status 429;
    }
}

Roll the limit out in observation mode first and measure how much real traffic it would have hit: an aggressive rule cuts off mobile carriers behind shared NAT addresses. Thresholds and algorithms are compared in the rate limiting guide.

Do not rely on User-Agent blocking as your primary defence: that string is changed by one option in any HTTP library. Rate limiting per address and per session targets the thing that actually costs money — request volume.

Symptom → quick check → likely cause → first action

Symptom	Quick check	Likely cause	First action
High load, wa near zero, us 90%+	ps aux --sort=-%cpu | head	Heavy process: PHP, cron, import, antivirus	Find and stop it, move the job to a queue
High load, high wa, low us	iostat -x 1, iotop -oPa	Disk bound: backup, unindexed query, IOPS cap	Postpone the background job, find the query in the slow log
Stuttering slowness, response time jitter	vmstat 1 (si/so), free -h	Swapping caused by RAM shortage	Reduce worker count, find the leak, add memory
Processes vanish without errors	dmesg -T | grep -i oom	OOM killer	Recalculate max_children and memory_limit
"No space left on device", disk looks free	df -i	Inodes exhausted	Purge the directory holding millions of small files
df says full, du cannot find the volume	lsof +L1	Deleted but still-open file	Reload or restart the holding process
502/504 while resources are idle	ss -ltn, PHP-FPM pool status	Workers or backlog exhausted	Check the upstream, raise workers within RAM budget
Connection spike, resources idle	ss -s, ss -tan state time-wait | wc -l	Bot wave or missing keep-alive	Parse access.log, enable rate limiting
Only one core out of eight is busy	top, press 1	Single-threaded workload	Parallelise or profile the code instead of adding cores
st in vmstat consistently above zero	vmstat 1	Steal: host oversubscription	Raise it with the provider, change plan or location

Quick fixes vs real fixes: why upgrading without diagnosis is the costliest path

Quick fixes exist to bring the site back right now. They do not address the cause, and they always require a return visit.

• Stop the background job that coincided with the peak: import, backup, reindexing.
• Enable or warm the page cache and serve anonymous visitors from it.
• Rate-limit the heavy URLs at the nginx level.
• Temporarily disable a heavy module or plugin if the spike started when it was enabled.
• Remove everything non-critical from cron for the next hour, and spread the rest across the minutes instead of firing everything on the hour.

Real fixes last: an index on the table, caching a result instead of recomputing it, moving long operations into a queue, profiling the slow code, compressing and offloading static assets, rewriting the query. The practical playbooks are in why a website loads slowly and website speed optimisation.

Adding resources is a legitimate option, but the last one on the list. If the cause is single-threaded code, more cores change nothing. If the cause is a missing index, you only postpone the problem until the table grows again — and you pay for the postponement every month. An upgrade is justified when diagnosis shows honest exhaustion of a resource in an already-optimised application: for example, a sustained high await pinned against the plan's IOPS cap. Then faster storage or a larger instance solves the problem predictably.

Rule: never change two parameters at once. One change, one measurement. Otherwise nobody on the team will be able to say a week later what actually helped and what merely coincided with a traffic dip.

Monitoring so you see it coming next time

One-off diagnosis answers "what is happening now". It does not answer "when did it start" or "what changed". That requires history. The minimum that pays for itself:

• System metrics retained for at least 30 days: load, CPU breakdown, memory and swap, space and inodes, IOPS and await, connection counts.
• Response time in the web server log, as shown above. Without $request_time you cannot tell "the server is slow" from "one URL is slow".
• Application layer: slow database queries, 5xx error rate, queue depth.
• Alerts on symptoms rather than resource thresholds: a rising 5xx share and rising response time matter more than "CPU above 80%". The reasoning is laid out in the golden signals of monitoring.
• Centralised logs with rotation — otherwise the post-mortem hits the wall of lines that were already deleted; see log management practices.

How to check from the outside

Internal diagnosis shows resources but not what the user sees. An external measurement closes that gap and also tells you whether the problem is on the server at all.

• Measure response time and TTFB. The site speed check reports time to first byte and full load time. A high TTFB with a fast remainder is exactly the pattern you were just chasing internally: the server is thinking too long.
• Record outages and degradation. Uptime monitoring gives you history: when it started, how long it lasted, whether it repeats on a schedule. Regular hourly peaks are nearly always cron or a backup.
• Separate your incident from someone else's. The outage tracker helps tell a local problem from an upstream one at a provider, CDN, or external API you depend on.
• Check whether spikes correlate with page errors. Mass 404s and redirect chains send crawlers in circles and generate pointless load — the broken link checker finds those routes.

Combine the two views: if the external measurement shows a stable TTFB while users still complain about slow pages, the problem is in the frontend, not in server load.

Frequently asked questions

What counts as normal server load?

There is no universal number. A workable guide: load average divided by core count staying below 0.7 means you have headroom; around 1 means running at the limit; above 2 means the queue is growing and response time is degrading. That said, high load with zero wa and fast page responses is not a problem — the server is busy but coping.

How do I check server load in Linux with one command?

vmstat 1 5 gives the most information per invocation: run queue, blocked processes, swap activity, I/O and the CPU time breakdown. If only top is available, press 1 for the per-core view and watch the %Cpu(s) line and the S state column.

How do I check free disk space in Linux?

df -h shows space and df -i shows inodes. Run both: a "full disk" with visibly free space is almost always exhausted inodes. To find what is consuming the volume, use du -xh --max-depth=1 / or the interactive ncdu -x /.

Should I disable swap on a web server?

No. Swap is not the cause of slowness, it is an indicator of memory shortage. Without swap, the kernel calls the OOM killer the moment RAM runs out and kills a process — usually the largest one, i.e. the database. A small swap area plus vm.swappiness=10 is the saner setup.

How can I tell bot load from real users?

Parse access.log: count requests per IP and per User-Agent and look at the share of static-asset requests. A real browser fetches CSS, fonts and images; a scraper takes HTML only. Sequential enumeration of every filter parameter combination, and dozens of requests per second from one address, are reliable signs of automation.

Will adding CPU and RAM help?

Only if diagnosis showed honest exhaustion of that specific resource. Extra cores go unused by a single-threaded job, and extra memory only postpones the pain of a missing index. Upgrading without diagnosis is the most expensive way to defer a fix — you pay for it every month.

Checklist: 15 minutes to the cause

• uptime and nproc — normalise load by core count, read the trend across the three numbers.
• vmstat 1 — identify the layer: r (CPU), b and wa (disk), si/so (swap), st (hypervisor).
• free -h — read available, not free; check whether swap usage is growing.
• dmesg -T | grep -i oom — check whether the OOM killer fired in the last few hours.
• df -h and, crucially, df -i — space and inodes on every filesystem.
• du -xh --max-depth=1 / or ncdu -x /; if it disagrees with df, run lsof +L1.
• iostat -x 1 — await and aqu-sz; iotop -oPa — who is reading and writing.
• ss -s and ss -ltn — connection counts and accept-queue overflow.
• ps aux --sort=-%cpu and --sort=-%mem — name the process.
• Logs: nginx response time, PHP-FPM slowlog, SHOW FULL PROCESSLIST and the database slow query log.
• access.log: top IPs, top User-Agents, static-asset share — separate bots from humans.
• External view: speed, history in monitoring, context in outages.
• Record one change, measure its effect, only then make the next one.