PROCESSES · CPUS · KERNEL · SCHEDULER · MEMORY · DISKS · LIMITS

Reading htop:
how a Linux server is put together

htop is a live view of one Linux machine: a bar for each CPU, the memory in use, and a row for every program running. Every one of those numbers is counted by a part of the kernel or the hardware, and once you know which part, you can tell what the machine is waiting for.

The server on this page is shop-1, a small cloud server with 4 CPUs and 16 GiB of memory that runs an online shop. nginx takes customers’ requests, the shop app builds each page, and PostgreSQL keeps the products and orders. Every night a backup copies the database to a compressed file. Later in the week a new release of the app starts leaking memory, so the last chapters can show their parts under real trouble.

Here is the whole server. Most boxes won’t mean much yet; each chapter explains one of them.

MapSolid outlines are real boundaries: the server, user space and the kernel. The dashed box is a cgroup, and arrows show what each part hands the next.

You need to know what a process and a file are; no kernel knowledge is assumed. The server is a made-up example running Ubuntu 26.04 LTS, Linux 7.0 and htop 3.4.1, the version Ubuntu 26.04 and Debian 13 install. Its numbers are invented but worked out the way Linux and htop compute them, so they add up from chapter to chapter; figures that show them carry an illustrative tag. htop’s colours are named in words, from its default scheme. Tap any figure to open it full screen and zoom. are tappable.

htop and /proc: every number is a file

htop is a terminal program that shows what one Linux machine is doing, redrawn every 1.5 seconds. On shop-1 it shows 4 CPU bars, two memory meters and 104 rows of numbers, one for each program and thread. None of it means much until you know where each number comes from and how old it is. Where does htop get its numbers, and what does one refresh measure?

STEP 01 / 04

THE MAP SO FAR

Fig 1.1The two accent boxes are htop and the files it reads; faint parts arrive in later chapters.

htop is an ordinary program in user space, the part of the machine where every program runs. It keeps no counters of its own. At each refresh it opens files under , which the kernel provides, and draws what it reads.

Every other box on the map is a part of the server those files report on: programs, CPUs, memory, disks, and the kernel parts that manage them. Each chapter adds one part and the htop numbers it produces.

STEP 02 / 04

/PROC FILES ARE WRITTEN AT THE MOMENT YOU READ THEM

Fig 1.2Follow 1 to 3: the text in the file didn’t exist until htop asked for it.

/proc is a filesystem with no disk under it. When a program reads /proc/loadavg, the kernel runs a small function that prints its current load counters as text, and that text is the whole file. ls -l lists these files at 0 bytes, because nothing is stored.

At every refresh htop reads a few machine-wide files: /proc/stat for CPU time, /proc/meminfo for memory, /proc/loadavg and /proc/uptime. Then, for every process and thread, it reads files such as stat and statm in its own directory, /proc/<pid>. On shop-1 that comes to about 410 files per refresh, in this tutorial’s illustrative numbers.

THREE /PROC FILES READ BY HAND (ILLUSTRATIVE OUTPUT)
# Three of the files htop reads every refresh, read by hand
cat /proc/loadavg
# 0.84 0.79 0.71 2/202 4410
#   load over 1, 5, 15 minutes · runnable/all tasks · newest PID

head -3 /proc/meminfo
# MemTotal:       16373760 kB
# MemFree:          317440 kB
# MemAvailable:    7147520 kB

grep '^cpu0 ' /proc/stat
# cpu0 1843502 1204 402118 10811936 20431 0 61209 0 0 0
#   clock ticks since boot: user nice system idle iowait irq softirq steal guest guest_nice
STEP 03 / 04

MOST NUMBERS ARE DIFFERENCES OVER 1.5 SECONDS

Fig 1.3The accent row is what the bar draws: the change between the two reads, not either read.

Most of the kernel’s counters only grow. The cpu0 line of /proc/stat holds the time cpu0 has spent in each mode since boot, in hundredths of a second. A total since boot says nothing about the last second.

So htop reads, waits and reads again. The wait is the refresh interval: 15 tenths of a second by default, set with htop -d. Each bar and each CPU% is the change between two reads over the time between them. In the figure, cpu0 was busy for 104 of 150 hundredths, so its bar reads 69.3%.

Memory, swap and the load average are levels, not counters, so htop shows each read as it is. The first frame is the exception: htop takes its first two reads a fraction of a second apart, so that frame’s bars and CPU% are rough.

STEP 04 / 04

THE HEADER IS THE MACHINE, THE LIST IS THE TASKS

Fig 1.4Each frame’s label names the /proc file it is drawn from; the accent frame is the process list.

The top of the screen, the header, describes the whole machine, and each meter comes from a machine-wide file. With 4 CPUs, htop puts every CPU bar in the left column, with Mem and Swp under them. Tasks, Load average and Uptime are text in the right column.

Below the header, each row is one process or thread, read from its own /proc/<pid> directory. The list is sorted by CPU% by default, busiest first. The next chapter explains what one row is, and the chapters after it explain the columns.

htop 3.4.1 has a second tab, I/O, which the Tab key switches to. It shows each process’s disk reads and writes, and chapter 08 uses it.

COMMON QUESTIONS

Does htop need root?
Not for the default screen. Root lets it show other users’ disk I/O and send signals to their processes.
Is htop different from top?
Both read the same /proc files, so the kernel’s numbers are the same. htop adds a bar per CPU, a process tree, scrolling and mouse support.
Does running htop slow the server?
A little. It reads files for every process at each refresh, so its cost grows with the number of processes. On shop-1 htop uses 1.2% of one CPU.

WHAT YOU NOW KNOW

  • htop reads files under /proc that the kernel writes at the moment of reading; it counts nothing itself.
  • Rates and percentages are the change between two reads 1.5 seconds apart, so the first frame and shorter spikes mislead.
  • The header comes from machine-wide files such as /proc/stat; each row comes from one /proc/<pid> directory.
DEEP DIVE: what htop reads per process, sampling and spikes, the same files in other tools, what changes in htop 3.5

What htop reads per process

At every refresh, htop reads stat (state, CPU time, priority) and statm (memory sizes) for each process and thread. It reads cmdline when a process first appears. Other files are read only while a column needs them: io for the I/O tab, smaps_rollup for PSS, oom_score for OOM and cgroup for CGROUP.

So htop’s own cost grows with the number of processes and threads, not with the load. On a host with thousands of threads, H hides the user threads and also skips most of their reads. Settings live in ~/.config/htop/htoprc, which htop saves when you change a setting in F2 and quit.

Sampling and spikes

A 1.5-second average hides short events. A stall of a few tens of milliseconds, a lock held too long or a burst of page faults, barely moves a bar. htop -d 5 refreshes every half second, at the price of more of htop’s own CPU.

For anything shorter, use a tool that records instead of redrawing. pidstat 1 prints each process’s CPU every second and keeps the history, and perf record samples the CPUs far more often and keeps every sample for later.

The same files in other tools

top reads /proc/stat, /proc/meminfo and each /proc/<pid>/stat, like htop. vmstat reads /proc/stat, /proc/meminfo and /proc/vmstat. free reads /proc/meminfo, uptime reads /proc/uptime and /proc/loadavg, and pidstat reads the files in each /proc/<pid>.

When two of these tools disagree, they almost always differ in the interval or in the formula, not in the data. Chapter 07 shows one such case: htop and free compute “used” memory differently from the same file.

What changes in htop 3.5

Upstream htop is at 3.5.3; Ubuntu 26.04 LTS and Debian 13 ship 3.4.1, which this tutorial teaches. In 3.5 the default columns replace SHR with PRIV (RES minus SHR), the CPU bar’s virt slice turns cyan instead of yellow, and the DiskIO meter splits into a rate meter and a time meter. htop --version prints which one you have.

Processes and threads: one row is one task

The list under htop’s meters on shop-1 is 104 rows long, yet the server runs only a handful of programs. nginx shows 5 rows, PostgreSQL 17 and the shop app 37. Before reading any column, you need to know what the kernel counts as one thing that runs. What is one row in htop, and why does each program have so many?

STEP 01 / 04

THE MAP SO FAR

Fig 2.1The accent boxes are the programs; in htop, each one becomes many rows.

The programs are the boxes in user space: nginx takes requests, shop builds pages, postgres stores the data, and at night backup copies the database. Each runs as one or more processes, and htop gives every process and thread its own row.

The kernel’s name for what each row shows is a : one thing it can run on a CPU. This chapter covers what makes a task, where tasks come from, and the columns that say who each one is.

STEP 02 / 04

TO THE KERNEL, A THREAD IS A TASK WITH ITS OWN PID

Fig 2.2Each accent row is one thread of the same worker, sharing the box’s memory.

A process is a group of tasks that share one memory: the same code, the same heap, the same open files. Each task in the group is a . Every thread has its own ID, and the group’s ID is the first thread’s, which the kernel calls the thread group ID (TGID).

A shop worker is one gunicorn process with 9 threads: a main thread that accepts connections, and 8 that each handle one request at a time. htop shows the process row, ID 905, then one row per other thread, with that thread’s own ID in the PID column.

Press H to hide the thread rows and see one row per process. The NLWP column, which you can add in F2, counts each process’s threads.

STEP 03 / 04

PROCESSES COME FROM FORK, AND THE TREE SHOWS WHO

Fig 2.3Follow 1 to 3 down the accent branch: PostgreSQL runs one process per helper and per connection.

Every process is started by another. The parent calls fork, which asks the kernel for a copy of the calling process, and the copy usually calls exec to load a different program. The parent’s ID becomes the new process’s PPID, so the processes form a tree under systemd, PID 1.

F5 in htop shows that tree. nginx’s master starts 4 workers and gunicorn’s master starts 4. PostgreSQL’s postmaster starts a helper process for each background job and a backend process for each client connection: 17 processes in all on shop-1.

PostgreSQL uses processes where gunicorn uses threads, so each connection in gunicorn’s pool shows up in htop as its own process, with its own PID.

POSTGRESQL’S PROCESS TREE (ILLUSTRATIVE OUTPUT)
# Every PostgreSQL process, as a tree, with its thread count (NLWP)
ps -o pid,ppid,nlwp,cmd --forest -C postgres
#    PID    PPID NLWP CMD
#   1020       1    1 /usr/lib/postgresql/18/bin/postgres -D /var/lib/postgresql/18/main
#   1021    1020    1   \_ postgres: checkpointer
#   1022    1020    1   \_ postgres: background writer
#   1023    1020    1   \_ postgres: walwriter
#   1024    1020    1   \_ postgres: autovacuum launcher
#   1025    1020    1   \_ postgres: logical replication launcher
#   1026    1020    1   \_ postgres: io worker 0
#   1027    1020    1   \_ postgres: io worker 1
#   1028    1020    1   \_ postgres: io worker 2
#   1104    1020    1   \_ postgres: shop shop 127.0.0.1(41882) idle
#   1105    1020    1   \_ postgres: shop shop 127.0.0.1(41890) idle
#    ... 6 more backends, one per connection in the pool
STEP 04 / 04

A ROW SAYS WHO, WHAT AND HOW MUCH CPU SO FAR

Fig 2.4Each column carries a chapter tag; the accent rows are the columns this chapter explains.

htop’s default columns, left to right, are PID, USER, PRI, NI, VIRT, RES, SHR, S, CPU%, MEM%, TIME+ and Command. Three of them say who a task is: PID is its ID, USER the account it runs as, and Command the command line it was started with.

TIME+ is the CPU time the task has used since it started, user and kernel time together, as minutes, seconds and hundredths. Its children’s time isn’t included. After 1 day, 12:30:04 of uptime, the postmaster has used 0:41.27; backend 1104, which runs the shop’s queries, has used 6:12.40.

The Tasks meter counts the same tasks: “Tasks: 58, 46 thr, 98 kthr; 2 running”. The kthr are kernel threads, which htop hides by default; K shows them, and chapter 04 explains them. Each of the other columns gets a chapter of its own.

COMMON QUESTIONS

What is a Z in the S column?
A zombie: the process has exited, but its parent hasn’t collected its exit status. It holds only a PID. Fix or restart the parent.
Why are some rows green?
Those are threads: htop draws thread rows in green by default. Press H to hide them and see one row per process.
What does F9 send?
SIGTERM by default, which a program can catch to shut down cleanly. SIGKILL can’t be caught, so the program gets no chance to clean up.

WHAT YOU NOW KNOW

  • Every row is a task; a thread is a task that shares its process’s memory, and the PID column shows its own ID.
  • A shop worker runs 9 threads; PostgreSQL runs one process per connection, which the tree (F5) shows.
  • TIME+ is CPU time used since the task started: 0:41.27 for the postmaster, 6:12.40 for a busy backend.
DEEP DIVE: clone and what threads share, zombies and orphans, signals from htop, PID limits

clone and what threads share

On Linux, starting a process and starting a thread are the same system call, clone, with different flags. A new process gets copies of its parent’s memory map, open files and signal handlers. A new thread is created with CLONE_VM, CLONE_FILES, CLONE_SIGHAND and CLONE_THREAD, so it shares all of them and joins its parent’s thread group.

Copying a process’s memory at every fork would be slow, so the kernel shares the parent’s pages read-only and copies a page only when one side writes to it (copy on write). That is why the postmaster can fork a backend for each new connection without copying its memory.

Zombies and orphans

When a process exits, the kernel keeps its exit status until the parent reads it with wait(). Until then the process is a zombie, Z in the S column: no memory, no CPU, only a PID. A parent that never calls wait leaves zombies behind.

If the parent exits first, the orphan is adopted by PID 1 or by the nearest subreaper, a process that asked to adopt its descendants, such as a systemd user manager. The new parent collects adopted zombies as they appear.

Signals from htop

F9 opens a list of signals, with SIGTERM selected. SIGTERM asks a program to exit, and it can clean up first. SIGKILL ends it at once and can’t be caught. SIGHUP makes many daemons reload their configuration, and SIGSTOP and SIGCONT pause and resume a process.

A gunicorn master watches its workers and starts a new one when one dies. When chapter 09’s OOM killer ends worker 907, the master starts worker 6127 in its place, so the row comes back with a new PID.

PID limits

Processes and threads take their IDs from one pool. On shop-1, /proc/sys/kernel/pid_max is 4,194,304, the value systemd sets on 64-bit machines. When the counter reaches it, the kernel wraps around and reuses free IDs, so a PID identifies a process only while it runs.

The limits that bite first are smaller: a cgroup’s pids.max, which systemd sets from each service’s TasksMax, and a user’s ulimit -u. A thread leak hits them long before pid_max, and the next fork or new thread fails.

ERRORS THIS CHAPTER EXPLAINS

fork: Resource temporarily unavailable
A process or thread limit was reached: a cgroup’s pids.max (a service’s TasksMax) or the user’s ulimit -u.
Find the thread leak (add the NLWP column and sort by it), or raise the limit.

Logical CPUs: a bar per CPU, a tick per mode

The bars at the top of htop’s screen show how busy each CPU is. shop-1 has 4, numbered 0 to 3, each filling and emptying with traffic. A bar at 63.3% is the most-read number in htop and one of the least understood. What is one bar, what does its percentage count, and how can one process use 296%?

STEP 01 / 04

THE MAP SO FAR

Fig 3.1The accent box at the bottom is the 4 vCPUs; each gets one bar in htop’s header.

The vCPUs are the parts that run instructions, one task at a time each. They sit in the hardware row, under the kernel, and everything a program does runs on one of them.

Each CPU the kernel can run a task on is a , and htop draws one bar for each. shop-1 has 4, numbered 0 to 3.

STEP 02 / 04

ONE BAR PER LOGICAL CPU, NOT PER CHIP

Fig 3.2All three show 4 bars; on the m7i.xlarge and shop-1, each accent core holds two of them.

A processor chip holds cores, and each core runs instructions. With simultaneous multithreading (SMT), one core runs two hardware threads, called on Intel, that share its execution units. Linux counts each hardware thread as a logical CPU.

On AWS, an Intel instance such as the m7i.xlarge gives you one hyperthread per vCPU, two per core. A Graviton instance such as the m7g.xlarge, or an AMD m7a, gives you a whole core per vCPU. Both show 4 bars, but the m7i.xlarge’s bars share 2 cores.

shop-1 is shaped like a t3.xlarge, an Intel instance, and lscpu reports 4 CPUs on 2 cores, 2 threads per core. Up to four CPUs, htop puts every bar in the left column; with five to eight it splits them over two columns, and above 128 it draws a single average bar.

STEP 03 / 04

THE KERNEL CHARGES EVERY TICK TO ONE MODE

Fig 3.3The accent slice is user time; each slice’s label gives the colour htop draws it in.

A CPU is always in exactly one mode: running a program (user), running kernel code (system), handling hardware (irq and softirq), or idle. The kernel counts CPU time in , 100 a second in /proc/stat, and adds each tick to the counter of the mode the CPU was in.

Each CPU’s line has a counter per mode: user, nice, system, idle, iowait, irq, softirq, steal and two guest counters. htop reads them twice, 1.5 s apart, and each slice of the bar is one mode’s share of the 150 ticks in between.

The default bar draws four slices: nice in blue, user in green, kernel (system, irq and softirq) in red, and virt (steal and guest) in yellow. Idle and iowait are drawn empty, and the number at the end is the sum of the rest. On cpu1 at 14:00, user took 52.0% and kernel 11.3%, so the bar reads 63.3%.

CPU1’S COUNTERS READ TWICE (ILLUSTRATIVE OUTPUT)
# What htop does every refresh: read the counters, wait 1.5 s, read again
grep '^cpu1 ' /proc/stat; sleep 1.5; grep '^cpu1 ' /proc/stat
# cpu1 1801377 1188 371902 10906016 19804 0 40113 0 0 0
# cpu1 1801455 1188 371916 10906070 19805 0 40116 0 0 0
#
# change: user 78, nice 0, system 14, idle 54, iowait 1, softirq 3: 150 ticks in all
# bar:    user 78/150 = 52.0% (green), kernel (14 + 3)/150 = 11.3% (red)
#         the number at the end of the bar: 63.3%
STEP 04 / 04

A PROCESS’S CPU% IS COUNTED AGAINST ONE CPU

Fig 3.4The accent bar is one process using several CPUs’ worth of time; each dashed mark is one CPU.

A process’s CPU% is its user and system ticks in the interval divided by one CPU’s ticks in the same interval. A single-threaded process tops out at 100%. A process with several threads can run on several CPUs at once, so its CPU% can pass 100%.

During Tuesday’s 02:00 backup, zstd, run with -T4 for four compression threads, reached 296%. pg_dump used 24%, and the PostgreSQL backend reading the tables for it 41%. The ceiling on shop-1 is 100% for each of its 4 CPUs: 400%.

The NCPU% column, which you can add in F2, divides by the CPU count instead, so 100% there means every CPU is busy.

COMMON QUESTIONS

Why don’t the rows’ CPU% add up to the bars?
Rows show only user-space tasks by default, and interrupt work belongs to no row. Processes that started and exited between two reads are missing too.
Is there an Irix mode toggle?
Not in htop; that is top’s. htop always counts against one CPU and offers the NCPU% column for the other view.
Does a bar at 50% mean half speed?
No. It means busy half the time. The clock speed can change too, and F2 can show each CPU’s frequency.

WHAT YOU NOW KNOW

  • shop-1 shows 4 bars, one per logical CPU; on Intel cloud instances, two bars share one core.
  • Each bar splits the last 1.5 s of clock ticks by mode: blue nice, green user, red kernel, and idle drawn empty.
  • CPU% counts against one CPU, so zstd’s 296% used nearly three of shop-1’s 4 CPUs.
DEEP DIVE: ticks and tickless kernels, SMT and real capacity, frequency, which CPU a task ran on

Ticks and tickless kernels

The hundredth of a second is only the unit /proc/stat reports in. The kernel keeps CPU time in nanoseconds and converts it when you read the file. With a tickless kernel (NO_HZ), an idle CPU stops its timer tick, and the kernel adds up the idle time when the CPU wakes.

A process’s split between user and system time is estimated from samples taken at the timer tick, then scaled to its exact total run time. That is why the sum of every process’s time never matches the CPU totals to the tick.

SMT and real capacity

Two hyperthreads share one core’s execution units, so together they get more done than one thread alone, and far less than two cores. Eight bars at 50% can be four cores fully busy, with little room left.

The scheduler knows which logical CPUs are siblings and prefers an idle core over the idle sibling of a busy one. Two CPU-heavy tasks on separate cores finish sooner than the same two on one core.

Frequency

A bar shows time, not work. A CPU busy half the time at a low clock speed does less than one busy half the time at full speed. htop can print each CPU’s frequency next to its bar: F2, Display options, “Also show CPU frequency”.

Which CPU a task ran on

The PROCESSOR column, whose header reads CPU, shows the CPU a task last ran on. The scheduler moves tasks between CPUs as the load changes. To keep a task on some CPUs, set its affinity with taskset -p or with htop’s a key.

User mode and kernel mode: system calls and interrupts

shop-1’s 4 bars are mostly green, user time, with a red slice in every one. Your code doesn’t run in the kernel, yet nginx’s workers spend most of their CPU time there. Something on the server spends CPU on behalf of your programs, and something else spends it on behalf of the network card. Why is part of every bar red, and who is that time spent for?

STEP 01 / 04

THE MAP SO FAR

Fig 4.1The accent band is the only way from a program into the kernel; the accent pair below is how hardware gets in.

This chapter adds the two ways into the kernel. The system call interface sits between user space and the kernel: every program that reads a socket, writes a file or starts a thread goes through it.

The other way in starts in hardware. The network card, or NIC, signals the kernel when packets arrive, and the kernel’s interrupt handlers run when it does. The NIC’s name depends on the machine, for example ens5 on shop-1.

STEP 02 / 04

A SYSTEM CALL RUNS KERNEL CODE ON THE CALLER’S TIME

Fig 4.2Follow 1 to 4: the accent boxes run kernel code; the bars below show that time charged to each worker.

A is a program asking the kernel to do something it isn’t allowed to do itself, such as read a socket or write a file. The CPU has two privilege levels, user mode and kernel mode: rings 3 and 0 on shop-1’s x86-64 CPUs, EL0 and EL1 on arm64. A system call switches to kernel mode, runs kernel code and switches back.

The kernel charges that time to the process that made the call, as its system time. htop adds it into the row’s CPU% and TIME+, and the CPU bars draw it red. One request to a shop worker makes two calls: recv() copies the request in, Python builds the page in user mode, and send() hands the page back to the kernel.

At Tue 14:00, 18% of a shop worker’s CPU time is system time. An nginx worker does little but move bytes (accept, read, sendfile, epoll_wait), so 71% of its time is system time.

STEP 03 / 04

INTERRUPTS RUN FOR NOBODY IN PARTICULAR

Fig 4.3Follow 1 to 5: the accent hops run on CPU 2, and no process pays for them.

An is a signal from hardware that makes a CPU stop what it is running and jump to a kernel handler. When a packet arrives, the NIC raises one. The handler runs for microseconds (irq time) and leaves the heavier TCP/IP work to a , which runs soon after on the same CPU (softirq time).

Neither is charged to a process. The task that was running is only paused, and the time goes to the CPU’s irq and softirq counters in /proc/stat. So no row in htop shows it; only the bars do.

shop-1’s NIC has 2 queues, each with its own interrupt line: ens5-Tx-Rx-0 goes to CPU 0 and ens5-Tx-Rx-1 to CPU 2. At Tue 14:00 they raise 2,770 interrupts a second between them, and NET_RX softirqs run 5,900 times a second. With Detailed CPU time on, cpu2 shows irq 1% in yellow and softirq 6% in magenta.

THE NIC’S INTERRUPTS AND NET_RX SOFTIRQS PER CPU (ILLUSTRATIVE OUTPUT)
# Interrupts per CPU since boot, for the network card ens5
grep -E 'CPU|ens5' /proc/interrupts
#             CPU0       CPU1       CPU2       CPU3
#  27:  171953210          0          0          0  PCI-MSIX-0000:00:05.0  1-edge  ens5-Tx-Rx-0
#  28:          0          0  191845116          0  PCI-MSIX-0000:00:05.0  2-edge  ens5-Tx-Rx-1

# Softirqs per CPU: NET_RX is the receive work each interrupt leaves behind
grep -E 'CPU|NET_RX' /proc/softirqs
#                    CPU0       CPU1       CPU2       CPU3
#       NET_RX:  402118734      21884  461205318      19022
STEP 04 / 04

KERNEL THREADS ARE HIDDEN UNTIL YOU PRESS K

Fig 4.4The accent row is the thread that takes over network work when interrupts arrive faster than CPU 2 handles them.

Some kernel work runs as tasks of its own. A runs only kernel code, has no user memory of its own, and is scheduled like any other task. htop hides them by default; press K to list them. The Tasks meter counts them anyway: 98 kthr on shop-1.

Their names say their jobs. ksoftirqd/N runs the softirqs CPU N couldn’t finish right after its interrupts. kworker threads run deferred kernel work, kswapd0 frees memory (chapter 09) and jbd2/nvme0n1p1-8 writes the ext4 journal.

Most of them wait. 90 of shop-1’s 98 kernel threads are in state I, idle, and the load average leaves those out (chapter 05). At Tue 14:00 the busiest is ksoftirqd/2 at 0.7%, on the CPU that takes the most of the NIC’s interrupts.

COMMON QUESTIONS

Is red time bad?
Not by itself. A web server spends much of its time moving bytes, which is kernel work. It is worth a look when it grows while traffic doesn’t.
Can I see which system calls a process makes?
Select it and press s: htop runs strace on it. strace slows the traced process a lot, so use it briefly.
Why is all the softirq on one or two CPUs?
Each NIC queue’s interrupt goes to one CPU. With fewer queues than CPUs, those few CPUs do most of the network work.

WHAT YOU NOW KNOW

  • A system call switches the CPU to kernel mode and charges the time to the caller: 71% of an nginx worker’s CPU time.
  • Interrupts and softirqs belong to no process, so only the bars show them, in their own colours once Detailed CPU time is on.
  • Kernel threads such as ksoftirqd and kswapd are tasks too, 98 on shop-1, hidden until you press K.
DEEP DIVE: what a system call costs, interrupt affinity, softirq types, tracing a process

What a system call costs

A system call costs two mode switches plus the kernel’s own work. The switches alone take hundreds of CPU cycles, more with the mitigations for speculative-execution bugs such as Meltdown. A few calls skip them: clock_gettime and gettimeofday usually run in the vDSO, a small piece of kernel code mapped into every process, so reading the clock stays in user mode.

A program that makes many small calls, such as reading a socket a few bytes at a time, shows a lot of red. Fewer, larger calls help: nginx’s sendfile moves a file to a socket in one call, without copying it through user space.

Interrupt affinity

/proc/irq/<N>/smp_affinity_list says which CPUs may take interrupt N, and /proc/interrupts counts where each one actually went. irqbalance, when it runs, moves interrupts between CPUs every few seconds to spread the load.

A NIC with several queues gives each queue its own interrupt, so receive work spreads over that many CPUs. With fewer queues than CPUs, receive packet steering (RPS) can hand the softirq work to other CPUs in software, set per queue in /sys/class/net/<nic>/queues/rx-<n>/rps_cpus.

Softirq types

/proc/softirqs has one row per type and one column per CPU: HI, TIMER, NET_TX, NET_RX, BLOCK, IRQ_POLL, TASKLET, SCHED, HRTIMER and RCU. NET_RX and NET_TX are network receive and send, BLOCK finishes disk I/O, TIMER and HRTIMER run expired timers, SCHED balances tasks between CPUs, and RCU frees memory once no reader can still see it.

The counts only grow, like every counter htop reads. Read the file twice and subtract to get a rate per CPU, which shows whether one CPU is doing all the network work.

Tracing a process

In htop, select a row and press s to run strace on it, or l to run lsof and list its open files and sockets. perf top shows which functions, kernel or user, are using the CPUs right now across every process, which is how you find the kernel code behind a lot of red.

Since Linux 6.1, /proc/pressure/irq reports the share of time tasks lost to interrupt work, on kernels built with IRQ time accounting. htop 3.4.1 can show it as the “PSI full IRQ” meter; chapter 05’s deep dive explains pressure stall information.

The scheduler: who runs, who waits

shop-1 has 4 CPUs and 202 tasks. Almost all of them are asleep at any moment, but when the backup starts, more tasks want a CPU than there are CPUs. Something picks which task runs where, and how long the rest wait. htop shows its work in four places: the S column, the Tasks meter, the load average and the NI column. Who gets a CPU, and how do you see the ones waiting?

STEP 01 / 04

THE MAP SO FAR

Fig 5.1The accent box chooses which task each CPU runs next; its arrow hands that task to a CPU.

The is the part of the kernel that picks which task runs on each CPU, and for how long. It keeps one run queue per CPU, the list of tasks ready to run there, and moves tasks between queues when one CPU is busier than another.

It sits in the kernel between the tasks of chapter 02 and the 4 CPUs of chapter 03. Every busy slice in chapter 03’s bars is time the scheduler gave to some task.

STEP 02 / 04

A TASK IS RUNNING, RUNNABLE OR ASLEEP

Fig 5.2Follow 1 to 5; the accent state is the only one that wants a CPU.

htop’s S column is the kernel’s state for each task. R means it wants a CPU: it is running on one, or waiting in a run queue. S means asleep until an event, such as a packet or a timer. D is asleep too, waiting for the disk, and no signal can wake it (chapter 08). Z is a zombie: it exited, and its parent hasn’t collected its exit status yet.

T and t are stopped, by a signal or by a debugger, and I is an idle kernel thread. htop prints the kernel’s parked state, P, as B.

At Tue 14:00, 0.8 tasks were R on average and 102 were S. The Tasks meter’s “running” stops at the CPU count: 1 m 50 s into Tuesday’s backup, 7 tasks were runnable and htop still showed 4 running.

STEP 03 / 04

LOAD AVERAGE COUNTS TASKS, NOT PERCENT

Fig 5.3The accent bar is the last minute; the dashed line at 4 is where every CPU would be busy.

The is a count of tasks, not a percentage. Every 5 s the kernel counts the tasks in state R or D and folds that count into three averages that fade over roughly 1, 5 and 15 minutes. /proc/loadavg prints them, and htop copies them into its Load average meter.

Read it against the CPU count. On shop-1’s 4 CPUs, a load of 4 keeps every CPU busy with nothing waiting; above that, tasks queue. At Tue 14:00 the load is 0.84 / 0.79 / 0.71.

Tuesday’s backup starts from 0.30. 1 m 50 s in, the 1-minute average is 4.17, above the CPU count, while the 5- and 15-minute averages are only 1.72 and 0.81. Read together, the three say the load is new and still rising.

/PROC/LOADAVG AND HTOP AT THE BACKUP’S PEAK (ILLUSTRATIVE OUTPUT)
# Tue 02:01:50, 1 m 50 s into the backup
cat /proc/loadavg
# 4.17 1.72 0.81 7/210 3880
#   1, 5 and 15 minute averages · 7 runnable of 210 tasks · newest PID
#
# htop at the same moment:
#   Tasks: 62, 50 thr, 98 kthr; 4 running       <- capped at the 4 CPUs
#   Load average: 4.17 1.72 0.81
STEP 04 / 04

NICE CHANGES THE SHARE, NOT THE SPEED

Fig 5.4The accent bar is the shop thread’s share while both threads want the same CPU.

Since Linux 6.6 the scheduler for normal tasks is EEVDF. It gives each runnable task CPU time in proportion to a weight, set by the task’s , from −20 to 19. Each step changes the weight about 1.25 times: nice 0 weighs 1,024 and nice 10 weighs 110.

htop’s NI column is the nice value and PRI is 20 + NI, so the backup’s pg_dump and zstd show NI 10 and PRI 30, and the bars draw their CPU time blue. When a shop thread and a zstd thread want the same CPU, they split it 90.3% to 9.7%.

With no traffic, nothing competes, and zstd takes every idle CPU at full speed. That is how Tuesday’s backup filled the bars: the CPUs were 97% busy, and 82% of that busy time was nice.

COMMON QUESTIONS

Is a load of 6 bad on 4 CPUs?
It means about 6 tasks wanted a CPU or the disk. If the bars are full, work is queuing for CPUs; if they aren’t, look for D in the S column.
What does RT in the PRI column mean?
A real-time task at the highest priority. Real-time tasks run before every normal task, and htop shows the lower real-time levels as negative PRI values.
Can a negative nice value make a process faster?
Only against other busy tasks on the same CPUs, and setting a nice value below 0 needs root.

WHAT YOU NOW KNOW

  • Only R tasks compete for a CPU; htop’s “running” stops at 4, while /proc/loadavg showed 7 runnable during the backup.
  • Load average counts R and D tasks over about 1, 5 and 15 minutes: 4.17 at the backup’s peak, against 4 CPUs.
  • Nice 10 gives the backup about a tenth of a contested CPU and all of an idle one.
DEEP DIVE: EEVDF, the load average arithmetic, pressure stall information, real-time and context switches

EEVDF

EEVDF, earliest eligible virtual deadline first, replaced CFS in Linux 6.6. It tracks each task’s lag: the CPU time it was owed minus the time it got. A task with a lag of zero or more is eligible, and among eligible tasks the one with the earliest virtual deadline runs next.

Virtual time runs slower for a heavier task, so its deadlines come round more often. The weights come from one table, sched_prio_to_weight, built so that one nice step moves a task’s share by about 10% against a task one step away. A nice 0 thread’s deadlines come round about nine times as often as a nice 10 thread’s.

The load average arithmetic

Every LOAD_FREQ, 5 s plus one tick, the kernel adds nr_running and nr_uninterruptible across all CPUs and updates each average: load = load × e + count × (1 − e). In fixed point, e is 1,884, 2,014 and 2,037 over 2,048 for the 1, 5 and 15 minute values, which are e to the power −5/60, −5/300 and −5/900.

So the 1-minute value covers about 63% of a step change within a minute, and the 15-minute value takes 15 minutes to do the same. Tasks in state I and frozen tasks don’t count, which is why shop-1’s 90 idle kernel threads add nothing.

The accent line follows the backup within a minute; the other two are still above it minutes after the backup ends.

The order of the three numbers tells the direction. A 1-minute value above the 15-minute value means the load is rising; below it, the load is falling, as it is a few minutes after the backup.

Pressure stall information

Since Linux 4.20, /proc/pressure/cpu reports the share of time at least one runnable task was waiting for a CPU: the some line, averaged over 10, 60 and 300 seconds. Unlike load, it is a percentage of wall time, and it counts only waiting, not running.

During Tuesday’s backup, some avg10 reached 38.40: for 38.40% of those 10 seconds, at least one task on shop-1 was ready to run and had no CPU. htop 3.4.1 can show it as the “PSI some CPU” meter (F2, Meters). The memory and io files work the same way, and every cgroup has its own.

Real-time and context switches

SCHED_FIFO and SCHED_RR tasks are real-time: they run before every normal task, highest priority first. A FIFO task runs until it blocks or yields; RR tasks of equal priority take turns. By default the kernel keeps 5% of each second for normal tasks, so a runaway real-time task can’t freeze a CPU completely.

/proc/<pid>/status counts each task’s context switches, the CPU changing from one task to another. A voluntary switch is the task blocking, say in recv(); a nonvoluntary one is the scheduler preempting it. A shop worker shows 1,284,113 voluntary and 41,027 nonvoluntary switches since it started. htop’s CTXT column shows their sum per refresh.

Virtual memory: VIRT, RES and SHR

Chapter 05 showed which task gets a CPU. Memory is the other thing every task needs, and htop gives each row four columns for it: VIRT, RES, SHR and MEM%. A shop worker shows about a gigabyte under VIRT and under a hundred megabytes under RES. The postgres rows together seem to hold more memory than shop-1 has. Neither is a bug: each process sees memory through a map the kernel keeps for it. Why is VIRT so much bigger than RES, and why can’t you add RES up?

STEP 01 / 04

THE MAP SO FAR

Fig 6.1The accent pair is new: the page tables in the kernel and the MMU in every CPU.

A is the kernel’s map from the addresses one process uses to places in RAM. Every process has its own set, so the same address in two processes can point at different memory. This is .

The (memory management unit) is the part of each CPU that reads those tables on every load and store. The kernel points it at a process’s tables whenever that process gets the CPU. RAM itself, and what else the kernel keeps in it, is chapter 07.

STEP 02 / 04

EVERY ADDRESS IS TRANSLATED, ONE PAGE AT A TIME

Fig 6.2Follow 1 to 3 for a page that has a frame, and 4 to 6 for a page touched first.

The kernel maps memory in of 4 KiB. On every load and store the MMU splits the address into a page number and an offset, looks the page up in the process’s page table, and reaches the frame of RAM that holds it (1 to 3).

A page with no frame yet stops the instruction with a . The kernel’s fault handler takes a free frame, zeroes it, writes the entry, and the instruction runs again (4 to 6). That is a minor fault: no disk was read. Faults that read the disk are major faults, chapter 09.

So a program gets RAM page by page, as it touches memory, however much it asked for. htop counts the faults per process in the MINFLT and MAJFLT columns, which are off by default; add them under F2, Screens.

STEP 03 / 04

VIRT IS WHAT IS MAPPED, RES IS WHAT IS IN RAM

Fig 6.3The top bar is everything the worker has mapped, to scale; the accent bar is the part in RAM.

VIRT is every page the process has mapped, with a frame or without one. Worker 905 maps 1,020 MiB. Half of it, 512 MiB, is glibc’s malloc arenas: 8 reservations of 64 MiB for its threads to allocate from. Another 72 MiB is 9 thread stacks of 8 MiB each, reserved and mostly untouched.

(resident set size) is the pages that have a frame in RAM now: 84 MiB. Of that, 61 MiB is anonymous memory, the heap and stacks the worker has touched, and 23 MiB is pages of the Python binary and its libraries. MEM% is RES over MemTotal, 0.5%.

htop prints this row as VIRT 1020M, RES 86016 and SHR 23552. A memory column under 100,000 has no suffix and is in KiB; above that, htop switches to MiB and adds an M. So the RES of 86016 is 84 MiB.

THE KERNEL’S FIELDS BEHIND VIRT AND RES (ILLUSTRATIVE OUTPUT)
# The fields behind VIRT, RES and SHR for shop worker 905
grep -E '^(VmSize|VmRSS|RssAnon|RssFile|RssShmem|Threads)' /proc/905/status
# VmSize:    1044480 kB     VIRT: everything mapped
# VmRSS:       86016 kB     RES: pages in RAM now
# RssAnon:     62464 kB     heap, stacks and arenas it has touched
# RssFile:     23552 kB     the Python binary and libraries (part of SHR)
# RssShmem:        0 kB
# Threads:         9
STEP 04 / 04

SHR IS PAGES OTHER PROCESSES CAN HOLD TOO

Fig 6.4The accent bar is what PostgreSQL really holds; the grey one counts shared pages once per process.

SHR is the part of RES that other processes can map too: pages of files, and shared memory, a region several processes map at once. For worker 905 it is 23 MiB, all of it the Python binary and libraries that every worker maps.

PostgreSQL keeps its 4 GiB of shared buffers in shared memory, and every backend counts each buffer it has touched in its own RES. Added up, the 17 postgres rows come to 27.4 GiB, more than shop-1’s 16 GiB of RAM.

PSS (proportional set size) divides each shared page among the processes that map it, so it adds up: 4.9 GiB. htop shows it in the PSS column. From htop 3.5.1 the default columns show PRIV, RES minus SHR, in place of SHR.

COMMON QUESTIONS

Why doesn’t RES drop after my program frees memory?
The allocator usually keeps freed pages to reuse instead of returning them to the kernel. RES falls only when pages are unmapped, which glibc does for large blocks and for free space at the top of the heap.
Can MEM% add up to more than 100%?
Yes, for the same reason as RES: a shared page counts in the MEM% of every process that maps it.
Why does a Go or Java process show a huge VIRT?
Their runtimes reserve large address ranges up front for the heap and thread stacks. A reservation costs no RAM until its pages are touched.

WHAT YOU NOW KNOW

  • The MMU translates every address through the process’s page table, one 4 KiB page at a time; a first touch costs a minor fault.
  • Worker 905 maps 1,020 MiB but holds 84 MiB in RAM: VIRT is what is mapped, RES what is resident.
  • PostgreSQL’s RES adds up to 27.4 GiB but its PSS to 4.9 GiB: a shared page counts in every process that maps it.
DEEP DIVE: overcommit, copy-on-write after fork, huge pages and the TLB, PSS and USS

Overcommit

vm.overcommit_memory defaults to 0, a heuristic: the kernel refuses only an allocation that obviously can’t fit, and otherwise grants address space without setting RAM aside. That is why malloc almost never fails on Linux. A shortage shows up later, when the pages are touched, as reclaim, swap and the OOM killer of chapter 09.

Mode 2 counts every promise against CommitLimit (swap plus vm.overcommit_ratio, 50% of RAM by default) and fails allocations up front instead. Committed_AS in /proc/meminfo shows how much has been promised in either mode.

Copy-on-write after fork

fork() copies a process’s page tables, not its pages. Parent and child map the same frames, marked read-only, and the first write to one of them faults; the kernel then copies that page alone. gunicorn’s --preload loads the app in the master before forking the workers, so pages they never write stay shared.

Those pages are anonymous memory, so they count in each worker’s RES but not in SHR; PSS splits them. In Python the sharing fades: reference counts live inside the objects, so touching an object writes its page and the kernel copies it.

Huge pages and the TLB

Walking the page table on every access would be slow, so each CPU caches recent translations in its TLB (translation lookaside buffer). With 4 KiB pages, a TLB of a few thousand entries covers only a few MiB, and a program that roams a large heap misses it often.

A 2 MiB huge page covers 512 times as much per entry. Transparent huge pages (THP) let the kernel use them for anonymous memory without the program asking; /sys/kernel/mm/transparent_hugepage/enabled sets the policy, and AnonHugePages in /proc/meminfo shows how much is in use.

PSS and USS

/proc/<pid>/smaps_rollup sums a process’s mappings: Rss, Pss, and the shared and private pages, clean and dirty. USS (unique set size) is the private pages alone: what ending the process would free. PSS is USS plus each shared page divided by the number of processes that map it.

Reading another user’s smaps needs ptrace access, so htop fills the PSS column for other users’ processes only when it runs as root. Parsing smaps costs time, so htop reads it for each process every second refresh.

RAM and the page cache: the Memory bar

Chapter 06 showed what each process holds. The Memory bar shows all of shop-1’s RAM, and on Tuesday afternoon it is nearly full of colour while MemFree, RAM holding nothing, is 310 MiB. Nothing is wrong. Linux keeps file data in RAM that programs aren’t using and gives it back when they need it. The bar tells those kinds of memory apart, if you know its colours. Is a server with no free memory in trouble, and how do you tell?

STEP 01 / 04

THE MAP SO FAR

Fig 7.1The accent pair is new: RAM, and the page cache the kernel keeps in it.

RAM is the machine’s memory: 16 GiB on shop-1, of which the kernel reports 15,990 MiB as MemTotal after its own code and reserved areas. It holds the pages of chapter 06, the kernel’s own data, and copies of files.

The is those copies: file contents the kernel keeps in RAM that programs aren’t using, so the next read skips the disk. On the map it sits in the kernel, joined to RAM, where its pages live.

STEP 02 / 04

THE MEMORY BAR HAS FIVE COLOURS

Fig 7.2The accent slice is file data; each label gives htop’s colour for that kind of memory.

htop builds the Memory bar from /proc/meminfo. Used (green) is MemTotal minus MemFree, Cached, SReclaimable and Buffers: memory programs hold. Shared (magenta) is Shmem, here mostly PostgreSQL’s shared buffers. Buffers (blue) is cached disk metadata. Cache (yellow) is Cached plus SReclaimable minus Shmem: file data and kernel caches it can drop.

On Tuesday at 14:00 that is used 4,220 MiB, shared 4,180 MiB, buffers 140 MiB, cache 7,140 MiB and free 310 MiB, out of 15,990 MiB. With zswap on, a grey compressed slice appears too.

The text at the end of the bar is used plus shared over the total: 8.20G/15.6G. The cache, the largest slice, is left out of the number.

THE MEMINFO LINES BEHIND THE BAR (ILLUSTRATIVE OUTPUT)
# The lines the Memory bar is computed from, in the order the file lists them
grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|Shmem|SReclaimable):' /proc/meminfo
# MemTotal:       16373760 kB
# MemFree:          317440 kB
# MemAvailable:    7147520 kB
# Buffers:          143360 kB
# Cached:         11161600 kB
# Shmem:           4280320 kB
# SReclaimable:     430080 kB
#
# The Memory bar in htop, in MiB:
#   used   = 15,990 - 310 - 10,900 - 420 - 140 = 4,220 MiB (green)
#   shared = Shmem = 4,180 MiB (magenta)
#   cache  = 10,900 + 420 - 4,180 = 7,140 MiB (yellow)
#   text   = (used + shared) / total = 8.20G/15.6G
STEP 03 / 04

A READ GOES TO THE PAGE CACHE FIRST

Fig 7.3Follow 1 and 2 for a cached page; the faint lane 3 is a miss, which chapter 08 explains.

Every read() of a file goes through the page cache. If the page is there, a hit, the kernel copies it from RAM into the program’s buffer (1, 2). If not, a miss, the kernel reads it from the disk, keeps it in the cache and then copies it (3, chapter 08).

PostgreSQL looks in its own shared buffers first and calls read() for a page that isn’t there. Monday’s backup ran soon after a reboot at Mon 01:30, with the cache empty, so every page missed. It took 4 m 45 s, and the cache grew by 6.0 GiB, from 1,020 MiB to 7.0 GiB.

Tuesday’s backup read the same database with its pages still cached, and took 1 m 50 s. A hit costs only the CPU time of the copy, so the backend never waits for the disk.

STEP 04 / 04

FREE MEMORY IS NOT AVAILABLE MEMORY

Fig 7.4The accent bar is what a new program could get without swapping; all four share one scale.

MemFree is RAM holding nothing at all: 310 MiB. is the kernel’s estimate of what a new program could get without swapping: free RAM plus the cache it can drop. It has been in /proc/meminfo since Linux 3.14. On shop-1 it is 6,980 MiB.

Tools disagree about used, too. htop’s used, 4,220 MiB, is programs’ own memory. free from procps-ng 4.0.7 prints MemTotal minus MemAvailable, 9,010 MiB, which also counts the shared buffers and memory the kernel can’t drop.

When MemAvailable runs low, the kernel drops cache first and then moves program memory to swap. Chapter 09 follows that.

COMMON QUESTIONS

Should I drop the page cache?
No, except to time a cold read in a benchmark. The kernel drops clean cache by itself when programs need the RAM; dropping it by hand only makes the next reads slower.
What are buffers?
Cached metadata of block devices, such as filesystem structures read straight from the disk. They are usually small: 140 MiB on shop-1.
Why is PostgreSQL’s memory shown as shared?
Its shared buffers are shared memory, which the kernel counts as Shmem inside Cached. htop draws Shmem magenta and takes it out of the cache, because it can’t be dropped like file data.

WHAT YOU NOW KNOW

  • The Memory bar splits RAM into used, shared, buffers and cache; its number, 8.20G/15.6G, leaves the cache out.
  • The page cache keeps file data in spare RAM: Tuesday’s backup took 1 m 50 s instead of Monday’s 4 m 45 s.
  • On Tuesday MemFree was 310 MiB but MemAvailable 6,980 MiB: read MemAvailable, not MemFree.
DEEP DIVE: dirty pages and writeback, what reclaim drops first, slab memory, tmpfs

Dirty pages and writeback

A write() lands in the page cache too: the kernel copies the data into a cached page, marks it dirty and returns. Flusher threads write dirty pages to the disk later. Dirty and Writeback in /proc/meminfo show how much is waiting.

vm.dirty_background_ratio (10% of available memory) is where background writeback starts; at vm.dirty_ratio (20%) a writing process has to wait for it. fsync() forces a file’s dirty pages out, which PostgreSQL does for its WAL at commit and for data files at a checkpoint.

What reclaim drops first

The kernel keeps file pages and anonymous pages on separate active and inactive lists. Reclaim takes pages from the inactive ends. A clean file page is simply dropped, since the disk has a copy; a dirty one is written first; an anonymous page needs swap, chapter 09.

A page read twice moves to the active list, so a one-off scan such as the backup doesn’t push out pages the shop reads all day. vm.swappiness (default 60) sets how readily reclaim takes anonymous pages instead of file pages. Newer kernels can use a multi-gen LRU, several generations instead of two lists, with the same aim.

Slab memory

The kernel allocates its own small objects from slab caches: dentries and inodes for files it has looked up, network buffers and much more. SReclaimable is the part it can free under pressure, mostly dentries and inodes; SUnreclaim is the part in use. htop counts SReclaimable as cache: 420 MiB on shop-1.

slabtop lists the caches by size. An SUnreclaim that keeps growing while the workload stays the same is worth a look: it is kernel memory nothing will give back.

tmpfs

tmpfs keeps files in the page cache with no disk behind them: /dev/shm, /run, and /tmp where it is configured that way. Its pages count as Shmem, so htop draws them magenta, and the kernel can’t drop them, only move them to swap.

A large file left in /dev/shm shows up under buff/cache in free and stays in RAM until someone deletes it.

Disks and I/O wait: the block layer

The nightly backup starts half an hour after shop-1 rebooted (Mon 01:30) for a kernel update, so the page cache is empty and every table page has to come from the disk. The load average climbs to 5.26, yet the CPU bars are mostly empty and no row uses much CPU. The tasks raising the load are asleep, waiting for the disk, and htop’s default bars don’t draw that wait at all. Why is the load high while the CPUs idle, and which process is reading the disk?

STEP 01 / 04

THE MAP SO FAR

Fig 8.1The accent parts are the path a page-cache miss takes: through the block layer to the disk.

The is the part of the kernel that queues reads and writes for each disk. Every read the page cache can’t serve becomes a request in its queue, and the disk driver takes requests from there.

shop-1 has one disk, nvme0n1: a network volume limited to 25 MiB/s and 500 operations a second, reads and writes together. It holds the 6 GiB database. Chapter 07 showed what happens when its pages are cached; this chapter is what happens when they aren’t.

STEP 02 / 04

A TASK WAITING FOR THE DISK SLEEPS IN D

Fig 8.2Follow 1 to 5: the task is in D from the moment its request is queued until the disk answers.

A task that needs a page the cache doesn’t hold can’t continue until the disk delivers it. The kernel queues a read request in the block layer and puts the task in , state D. Signals, even SIGKILL, wait until the request completes.

In Monday’s cold backup the reader is the backend serving pg_dump, PID 1200. It was in D in 72% of htop’s samples. The 8 backends serving the shop were in D 35% of the time each, their reads waiting in the same queue.

Chapter 05’s load average counts tasks in D as well as runnable ones. So the load reached 5.26 / 3.37 / 1.62 while the CPUs were only 31% busy.

TASKS IN D AND THE KERNEL FUNCTION EACH WAITS IN (ILLUSTRATIVE OUTPUT)
# Mon 02:03, during the cold backup: every task in D, and the kernel function it waits in
ps -eo pid,stat,wchan:22,cmd | awk '$2 ~ /^D/'
#  1200 D    folio_wait_bit_common  postgres: shop shop [local] COPY
#  1106 D    folio_wait_bit_common  postgres: shop shop 127.0.0.1(41896) SELECT
#  1109 D    io_schedule            postgres: shop shop 127.0.0.1(41918) SELECT
#
# Each is asleep inside the kernel until the disk answers; each counts in the load average.
STEP 03 / 04

IOWAIT IS IDLE TIME, AND THE DEFAULT BAR HIDES IT

Fig 8.3Both bars are one CPU over the same 1.5 s; only the detailed bar draws the accent iowait.

is time a CPU had nothing to run while a task that last ran on it waited for I/O. The CPU isn’t waiting: any runnable task could use it. The kernel’s own documentation calls the number unreliable.

htop’s default bar counts iowait as idle and draws nothing for it. With Detailed CPU time on (F2, Display options, off by default) it draws iowait as a grey slice, but still leaves it out of the percentage.

During the cold backup each CPU averaged 31% busy and 27% iowait. Both bars print 31.0%: htop never counts the wait as work.

STEP 04 / 04

THE I/O TAB SHOWS WHO MOVES BYTES TO DISK

Fig 8.4The accent row is the process actually reading the disk; pg_dump only receives what it reads.

Press Tab and htop switches from the main screen to its I/O screen. Each row shows the process’s I/O priority (IO), its DISK READ and DISK WRITE rates and their sum, DISK R/W, which the screen sorts by. SWPD% and IOD% need delay accounting (deep dive).

DISK READ and DISK WRITE come from read_bytes and write_bytes in /proc/<pid>/io: bytes that reached the block layer. A read served from the page cache never appears. Other users’ processes show N/A unless htop runs as root.

Monday’s backend read 21.6 MiB/s against the volume’s 25 MiB/s cap, and zstd wrote 2.7 MiB/s. On Tuesday, with the database cached, the backup’s backend read 0.0 MiB/s and the whole backup took 1 m 50 s instead of 4 m 45 s.

COMMON QUESTIONS

Why can’t I kill a D process?
It is inside the kernel waiting for a device, and only the request’s completion wakes it. A D state that lasts minutes points at a stuck disk or network filesystem.
Is high iowait bad?
It only says CPUs were idle while I/O was pending. Disk latency and queue length, from iostat -x, say whether the disk is the problem.
Why does pg_dump read nothing from disk?
The PostgreSQL backend reads the tables and sends rows to pg_dump over a socket. A socket isn’t a disk, so only the backend’s row moves.

WHAT YOU NOW KNOW

  • A read that misses the cache queues a request in the block layer, and the task sleeps in D until the disk answers.
  • Tasks in D raise the load: 5.26 while the CPUs were only 31% busy.
  • The default bar counts iowait as idle; the I/O tab shows the backend reading 21.6 MiB/s.
DEEP DIVE: latency and queue depth, I/O schedulers, dirty-page throttling, network filesystems, delay accounting

Latency and queue depth

/proc/diskstats counts each disk’s completed reads and writes and the time they took. iostat -x (from sysstat) turns two reads of it into rates: r/s, reads a second; r_await, the average milliseconds per read, queue included; and aqu-sz, the average number of requests in flight.

During the cold backup nvme0n1 served 390 reads a second at 18.4 ms each, so about 7.2 reads were waiting at any moment (r/s × r_await). The backend and zstd together moved 24.3 MiB/s, close to the 25 MiB/s cap, so requests waited their turn. On Tuesday the same disk saw 2 reads a second at 0.6 ms.

The accent bar is the cold backup’s time per read: the volume’s queue, not the CPUs.

htop’s Disk IO meter (F2, Meters; not in the default header) shows the machine’s read and write rates and how busy the disk is. htop 3.5 splits it into two meters, DiskIORate and DiskIOTime.

I/O schedulers

Each disk has a scheduler, chosen in /sys/block/nvme0n1/queue/scheduler. NVMe devices, including cloud volumes that appear as NVMe, default to none: requests go to the device in arrival order. mq-deadline and bfq are the alternatives.

The IO column shows a process’s I/O priority. With no class set, htop prints B and (nice + 20) ÷ 5, so the nice-10 backup reads B6. Those levels only change anything under bfq; with none, ionice has no effect.

Dirty-page throttling

A write normally lands in the page cache as a dirty page, and kernel flusher threads write it out later. When a process dirties pages faster than the disk takes them, balance_dirty_pages() makes it sleep for a moment on each write. It shows as D and raises the load like a reader does, though this sleep does end on SIGKILL. vm.dirty_ratio and vm.dirty_background_ratio set the limits.

Network filesystems

An NFS mount with the default hard option retries a request until the server answers. If the server goes away, every process touching the mount sleeps in D, and the load average climbs by one for each. On current kernels SIGKILL can end those waits; other signals can’t. A load that keeps rising with idle CPUs and no disk traffic often means this.

Delay accounting

IOD%, SWPD% and CPUD% are the share of time a task spent waiting for block I/O, for swap-in and for a CPU. htop reads them over a taskstats netlink socket, which needs CAP_NET_ADMIN and an htop built with --enable-delayacct. The kernel only counts them with kernel.task_delayacct=1 (off by default). Otherwise the columns show N/A or 0.0.

/proc/pressure/io gives the machine-wide view without any of that. Its some line is the share of recent time in which at least one task waited for I/O; its full line, the share in which every non-idle task did. htop shows both as the PSI some IO and PSI full IO meters.

ERRORS THIS CHAPTER EXPLAINS

INFO: task postgres:1200 blocked for more than 120 seconds.
The kernel’s hung-task check found a task in D for that long; its stack follows the line.
Check the device or the NFS server the task waits on, not the process itself.

Swap and the OOM killer: when memory runs out

On Wednesday release 2 of the shop app goes out, and every request now leaves 256 KiB behind in a worker: 50 MiB/s at 200 requests a second. The Memory bar’s green grows, the cache shrinks, and then the Swap bar, empty all week, starts to fill. At 10:03:02 one worker’s row disappears. What happens when memory runs out, and who decides which process dies?

STEP 01 / 04

THE MAP SO FAR

Fig 9.1The accent pair is where pages go when RAM is short, and what acts when there is nowhere left.

is disk space for program memory that has to leave RAM. On shop-1 it is the 2 GiB file /swap.img on nvme0n1, next to the database files.

(kswapd0 in the process list) is a kernel thread that frees RAM in the background. When freeing fails, the ends one process to get its memory back.

STEP 02 / 04

RECLAIM DROPS CACHE FIRST, THEN SWAPS

Fig 9.2Read top to bottom: the cache shrinks first, then the accent swap slice grows.

When free RAM falls below a watermark, kswapd wakes and reclaims pages. Clean page-cache pages go first: the file is still on disk, so the kernel just drops them. Program memory (anonymous pages) has no file behind it, so it must be written to swap before its frame can be reused.

vm.swappiness (default 60, range 0–200) weights that choice. Higher values make the kernel swap program memory sooner; lower ones make it drop more cache first.

At Wed 10:00:00 the cache held 7.0 GiB. 2 m 25 s later it was down to 180 MiB, and only then did swap start to fill: 1,530 MiB at +2 m 50 s.

STEP 03 / 04

A MAJOR FAULT WAITS FOR THE DISK

Fig 9.3Follow 1 to 5: the accent hop is a disk read inside what looked like a memory access.

A swapped-out page has no frame in RAM. When the program touches it, the MMU raises a page fault, the kernel reads the page back from the swap file through the block layer, and the task sleeps in D until it arrives. That is a major fault, counted in the MAJFLT column (F2, Columns).

At +2 m 50 s the shop workers took 1,840 major faults a second between them, and kswapd0 used 41% of a CPU. On the Swap bar, used is red and cache is yellow: pages that are in RAM and still have a copy in swap.

STEP 04 / 04

THE OOM KILLER KILLS THE BIGGEST SCORE

Fig 9.4The accent row had the highest score and was killed; the score is size, not blame.

When reclaim can’t free enough memory, the OOM killer picks one task and sends it SIGKILL. It picks by badness: the task’s resident pages, swap entries and page tables, shifted by oom_score_adj (−1,000 to 1,000).

htop’s OOM column (F2, Columns) shows /proc/<pid>/oom_score, badness rescaled against RAM plus swap. A tiny process reads 666, the floor, and the highest score dies.

At 10:03:02 leaking worker 907 scored 756, just above its siblings and PostgreSQL’s backend 1104 at 742. The kernel killed it and logged why, and the gunicorn master started worker 6127 in its place.

THE KERNEL’S OOM REPORT (ILLUSTRATIVE OUTPUT)
# The kernel log around the kill, Wednesday 10:03:02
journalctl -k --since 10:03 | grep -E 'invoked oom-killer|oom-kill:|Out of memory'
# gunicorn invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
# oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/shop.service,task=gunicorn,pid=907,uid=998
# Out of memory: Killed process 907 (gunicorn) total-vm:3412880kB, anon-rss:1986412kB, file-rss:6144kB, shmem-rss:0kB, UID:998 pgtables:5012kB oom_score_adj:0
#
# total-vm is VIRT; anon-rss + file-rss + shmem-rss is RES; all sizes in kB

COMMON QUESTIONS

Is any swap use bad?
No. Pages swapped out once and never touched again cost nothing. Major faults per second, not swap used, say whether swap hurts.
Should I turn swap off?
Without swap, program memory can never leave RAM, so the kernel drops more cache and reaches the OOM killer sooner.
Where is the record of a kill?
In the kernel log, from journalctl -k or dmesg: each candidate’s size, the victim’s name and its sizes at the kill.

WHAT YOU NOW KNOW

  • Under pressure kswapd drops cache first, then writes program memory to swap.
  • Touching a swapped page is a major fault, a disk read: 1,840 a second at the worst.
  • The OOM killer killed the worker with the highest score, 756; the score measures size, not blame.
DEEP DIVE: swappiness, zswap and zram, memory pressure, systemd-oomd and earlyoom

swappiness

vm.swappiness runs from 0 to 200 and defaults to 60. It is the relative cost the kernel assigns to swapping program memory against dropping file cache: 100 means equal, lower favours dropping cache. Values above 100 suit swap that lives in RAM, such as zram. At 0 the kernel avoids swapping until free and file-backed pages run very low; it doesn’t turn swap off.

zswap and zram

The zswap cache compresses pages on their way to swap and keeps them in a pool in RAM, writing to the swap file only when the pool fills. The zram driver is a compressed block device in RAM, used as swap instead of a file. htop has a Zram meter for it.

The Swap bar’s grey segment shows what zswap holds. Its label still says “frontswap”, although the kernel removed frontswap in 6.5. htop 3.5 adds meters for the zswap pool itself.

Memory pressure

/proc/pressure/memory reports pressure stall information (PSI). some is the share of recent time in which at least one task stalled on memory (reclaim, a swap-in); full is the share in which every non-idle task did. At +2 m 50 s shop-1 read some avg10 62.10 and full avg10 38.40.

So for more than a third of those ten seconds nothing on the machine made progress. htop shows these numbers as the PSI some memory and PSI full memory meters (F2, Meters).

systemd-oomd and earlyoom

systemd-oomd and earlyoom are killers in user space. They watch pressure or free memory and kill a cgroup or a process before the kernel runs out, which is gentler than a stall followed by the kernel’s choice.

systemd-oomd isn’t installed on Ubuntu Server by default (it comes with the desktop), so on shop-1 the kernel’s OOM killer acted. Check a machine with systemctl is-active systemd-oomd.

Cgroups and the hypervisor: limits htop doesn’t draw

Everything so far shows up somewhere in htop. Two parts of shop-1 don’t: the cgroups that cap a group of processes, and the hypervisor that runs shop-1 itself on a cloud host’s hardware. Both can slow a process while htop shows idle CPUs and free memory. What can slow a process that htop can’t see, and where do you look instead?

STEP 01 / 04

THE MAP SO FAR

Fig 10.1The accent dashed band is the backup’s cgroup; the accent band at the bottom runs shop-1 itself.

A (control group) is a kernel group of processes with its own CPU, memory and I/O limits. systemd puts every service in one, such as shop.service, and the nightly backup runs in batch.slice. The scheduler and the memory code enforce the limits.

The is the software on the cloud host that runs shop-1’s 4 vCPUs as threads on its own cores. Neither part has a meter in htop.

STEP 02 / 04

A CPU QUOTA THROTTLES WITHOUT A BAR TO SHOW IT

Fig 10.2The accent bar is the same backup under a one-CPU quota.

A caps the CPU time a cgroup’s tasks get per period. CPUQuota=100% writes 100000 100000 to cpu.max: 100 ms of CPU time in every 100 ms period, one CPU’s worth, shared by every thread in the slice.

zstd’s worker threads spend that early in each period, then wait until the next one starts. On Tuesday the on-call engineer reran the backup under this quota: it took 5 m 52 s instead of 1 m 50 s, zstd showed 93%, and the CPU bars stayed mostly empty.

htop draws nothing for it: the threads just get less CPU, so their CPU% drops. Only the cgroup’s cpu.stat counts it: 3,447 of 3,520 periods throttled.

A ONE-CPU QUOTA AND ITS COUNTERS (ILLUSTRATIVE OUTPUT)
# Tue 02:30: rerun the backup with a quota of one CPU on its slice
sudo systemctl set-property --runtime batch.slice CPUQuota=100%
sudo systemd-run --slice=batch.slice --nice=10 /usr/local/bin/backup.sh

cat /sys/fs/cgroup/batch.slice/cpu.max
# 100000 100000       100 ms of CPU per 100 ms period: one CPU

# after the run: the four counters that matter here
grep -E '^(usage_usec|nr_periods|nr_throttled|throttled_usec) ' /sys/fs/cgroup/batch.slice/cpu.stat
# usage_usec 352000000
# nr_periods 3520
# nr_throttled 3447
# throttled_usec 256800000
STEP 03 / 04

INSIDE A CGROUP, HTOP STILL SHOWS THE WHOLE MACHINE

Fig 10.3The accent row is where they disagree: htop shows gigabytes available, the scope has none left.

htop builds its Memory, Swap and CPU meters from /proc/meminfo and /proc/stat, which describe the whole machine. Inside a container or a limited systemd scope, htop still shows shop-1’s 16 GiB and 4 CPUs.

On Wednesday afternoon the monthly report (report.py, PID 6480) ran in a scope with a 512 MiB memory limit. When it reached the limit, the kernel killed it inside the cgroup (“Memory cgroup out of memory”), while htop in the same scope showed 6.8 GiB available.

A MEMORY-CAPPED SCOPE AND ITS OOM KILL (ILLUSTRATIVE OUTPUT)
# Wed 15:00: the monthly report, in a scope capped at 512 MiB
sudo systemd-run --scope --unit=report -p MemoryMax=512M python3 report.py
# Running as unit: report.scope
# Killed
echo $?
# 137       128 + 9: the process ended on SIGKILL

journalctl -k --since 15:00 | grep 'Memory cgroup out of memory'
# Memory cgroup out of memory: Killed process 6480 (python3) total-vm:845312kB, anon-rss:515072kB, file-rss:9216kB, shmem-rss:0kB, UID:1000 pgtables:1240kB oom_score_adj:0

journalctl -u report.scope --since 15:00 | grep -i 'oom'
# report.scope: A process of this unit has been killed by the OOM killer.
#   systemd reads it from memory.events of the scope, where oom_kill went from 0 to 1
STEP 04 / 04

STEAL IS TIME THE HYPERVISOR GAVE TO SOMEONE ELSE

Fig 10.4The accent slice is time the vCPU wanted but the host didn’t give it.

Each vCPU is a thread on the cloud host. When it has work but the host runs something else, the guest kernel counts : the eighth value on each cpu line of /proc/stat. htop adds it to the virt segment, drawn yellow in 3.4.1 (the irq colour, easy to misread) and cyan from 3.5.

shop-1 is shaped like a t3.xlarge in standard credit mode. It earns CPU credits while it runs below its baseline of 40% per vCPU, spends them above it, and once they are gone is held to the baseline. T3 instances default to unlimited mode; this one was set to standard.

On Thursday night the credits ran out. Each bar showed user 33%, kernel 7% and steal 55%: the work that fit in the baseline ran, and the rest of the time went elsewhere. AWS re:Post guidance describes this: credit throttling on T-series instances appears as steal.

COMMON QUESTIONS

Does a Kubernetes CPU limit show in htop?
No. It becomes cpu.max on the pod’s cgroup, the same as this quota. Read cpu.stat or the container runtime’s metrics.
Can htop show cgroups at all?
The CGROUP column (F2, Columns) shows each process’s cgroup path, which is enough to tell containers and services apart.
Is steal possible on bare metal?
No. Without a hypervisor nothing takes a CPU away, so steal stays at zero.

WHAT YOU NOW KNOW

  • A one-CPU quota made the backup take 5 m 52 s with the CPUs mostly idle; only cpu.stat showed 3,447 throttled periods.
  • Inside a cgroup htop still shows the whole machine, so a 512 MiB limit ended in a kill htop didn’t predict.
  • Steal is runnable time the hypervisor spent elsewhere; htop draws it in the virt segment.
DEEP DIVE: the cgroup tree, quotas and periods, memory.high and memory.max, lxcfs, vCPUs on AWS

The cgroup tree

cgroup v2 is one tree mounted at /sys/fs/cgroup. systemd builds it from slices: system.slice holds the services, user.slice the login sessions, and shop-1’s batch.slice sits beside them. systemd-cgls prints the tree with each process under its unit, and htop’s CGROUP column shows the same path on each row.

Quotas and periods

The quota is enforced per 100 ms period. zstd’s four busy threads spend a one-CPU quota in a quarter of the period, then all wait for the rest of it. A request that arrives at the wrong moment waits that long too, which is why a quota hurts latency more than its average suggests.

Tuesday’s rerun lost 256.8 s to throttling in total (throttled_usec). cpu.max.burst lets a cgroup bank unused quota for later periods. Each cgroup also has cpu.pressure, memory.pressure and io.pressure, the PSI numbers of chapters 08 and 09 for that group only.

memory.high and memory.max

memory.max is the hard limit. At it the kernel reclaims inside the cgroup, and if that fails it kills a task in it, as it did to the report. memory.high is softer: above it the cgroup’s tasks are slowed and pushed into reclaim, but nothing is killed. memory.events counts high, max, oom and oom_kill; systemd sets the limits with MemoryHigh= and MemoryMax=.

lxcfs

lxcfs is a small FUSE filesystem that some container runtimes, such as LXD and Incus, mount over /proc/meminfo, /proc/stat and a few other files inside a container, with values computed from the container’s cgroup. htop inside such a container shows its own limit as the total. Docker and Kubernetes don’t use it by default.

vCPUs on AWS

On x86 Intel instances such as m7i and T3, a vCPU is one hyperthread, two per core. On Graviton (m7g, m8g) and on AMD m7a, a vCPU is a whole core.

Burstable T instances earn CPU credits below their baseline and spend them above it. In standard mode an instance out of credits is held to the baseline, 40% per vCPU on a t3.xlarge; in unlimited mode it keeps running and pays for the surplus. AWS re:Post guidance notes that the throttling shows as steal time.

ERRORS THIS CHAPTER EXPLAINS

Memory cgroup out of memory: Killed process …
The process’s cgroup reached its memory.max; the machine itself may have plenty free.
Raise the limit (MemoryMax=, a container’s memory limit) or fix the leak; memory.events counts the kills.

The whole picture

Here is shop-1 with every part numbered; the table below says what each number is.

Numbers match the table below. Read the server top to bottom: programs, the kernel, the hardware, the host.
  1. 1htop

    reads /proc every 1.5 s and draws the differences

    sits in
    user space
    htop shows
    the whole screen
    when wrong
    short spikes averaged away; a rough first frame
  2. 2/proc

    the kernel’s counters as files, written when read

    sits in
    the kernel
    htop shows
    every number
    when wrong
    nothing: it reports and doesn’t fail
  3. 3processes and threads

    the programs; each thread is a task

    sits in
    user space
    htop shows
    the rows, PID, USER, TIME+, the tree, Tasks
    when wrong
    Z rows; fork failures when the PID limit is reached
  4. 4logical CPUs

    run instructions; every tick is charged to a mode

    sits in
    hardware
    htop shows
    one bar each; CPU%
    when wrong
    full bars and more runnable tasks than CPUs
  5. 5system call interface

    the door from a program into kernel mode

    sits in
    between user space and the kernel
    htop shows
    the red share of the bars; system time in TIME+
    when wrong
    red growing while traffic doesn’t
  6. 6interrupts and softirqs

    handle hardware events, such as packets

    sits in
    the kernel
    htop shows
    irq and softirq (detailed CPU time); ksoftirqd
    when wrong
    one CPU busy with softirq; a busy ksoftirqd
  7. 7network card (NIC)

    receives packets and raises interrupts

    sits in
    hardware
    htop shows
    nothing directly
    when wrong
    interrupts piled on one CPU
  8. 8scheduler

    chooses who runs on each CPU, by weight

    sits in
    the kernel
    htop shows
    S column, Tasks running, load average, PRI and NI, blue time
    when wrong
    load above the CPU count with full bars
  9. 9page tables

    map each process’s pages to RAM

    sits in
    the kernel
    htop shows
    VIRT, RES, SHR, MEM%
    when wrong
    RES climbing and never falling
  10. 10MMU

    translates every address a program uses

    sits in
    inside each CPU
    htop shows
    minor and major page faults
    when wrong
    major faults every second
  11. 11RAM

    holds every page in use

    sits in
    hardware
    htop shows
    the Memory bar’s total
    when wrong
    MemAvailable near zero
  12. 12page cache

    keeps file data in spare RAM

    sits in
    the kernel, in RAM
    htop shows
    the Memory bar’s cache and buffers
    when wrong
    a cold cache: slow first reads and high DISK READ
  13. 13block layer

    queues reads and writes for the disk

    sits in
    the kernel
    htop shows
    D state, iowait, the I/O tab
    when wrong
    high load with idle CPUs; tasks in D
  14. 14disk

    stores the database files and the swap file

    sits in
    hardware
    htop shows
    DISK READ and DISK WRITE
    when wrong
    reads stuck at the volume’s limit
  15. 15swap file

    holds program memory moved out of RAM

    sits in
    the disk
    htop shows
    the Swap bar
    when wrong
    swap filling while major faults climb
  16. 16kswapd and the OOM killer

    free memory; kill a task when that fails

    sits in
    the kernel
    htop shows
    kswapd0’s CPU%, the OOM column
    when wrong
    a row that vanishes and a kernel log line
  17. 17cgroups

    cap a group’s CPU, memory and I/O

    sits in
    the kernel
    htop shows
    only the CGROUP column
    when wrong
    idle CPUs and a slow job; a kill with memory free
  18. 18hypervisor

    runs the VM’s vCPUs on real cores

    sits in
    under the VM
    htop shows
    steal time, in the virt segment
    when wrong
    a slice of every bar no process explains

THREE THINGS TO KEEP

  • htop counts nothing itself. Every bar and column is a kernel counter, compared between two reads.
  • Most slowness is waiting for a CPU, the disk or memory. A task’s state tells you which.
  • htop sees the whole machine, not the limits around a container. A throttled process can look idle.
Read next. Prometheus for keeping these numbers over time instead of watching them live, and Kubernetes for the cgroups of chapter 10 at the scale of a cluster.

next time a server is slow, read each number back to the part that counted it