CPU frequency governors on servers

5 min read

gentoo system-administration

A file server here spent several years running every build, scrub and resilver at 1.2 GHz on hardware rated for 2.5 GHz. Nothing was throttling it, nothing was overheating, and the machine reported its full clock range the whole time. The cause was one line in a hand-rolled kernel config:

CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y

The userspace governor does not scale. It holds whatever frequency was last written to scaling_setspeed, and on that box nothing ever wrote one, so the cores sat at the minimum P-state forever. One write to scaling_governor and busy cores climbed to 2.3 GHz.

This is a failure mode that hides well, because every individual number the system reports is correct. It is worth knowing how the pieces fit together.

Three layers decide your clock speed

Only one of them is the kernel governor, and it is often not the one in charge.

The driver is the bottom layer. Check it first:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
  • intel_pstate is the active mode of Intel's driver. The kernel hands a performance hint to the CPU and the hardware picks the actual P-state. Only performance and powersave are offered as governors here, and they are not the generic governors of the same name.
  • intel_cpufreq is the same driver in passive mode: it registers as an ordinary cpufreq driver, so the full generic governor list becomes available.
  • acpi-cpufreq is the generic ACPI driver, used on older Intel parts and on AMD without amd-pstate.

The governor is the policy on top of the driver:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

performance pins to the maximum, powersave to the minimum, schedutil drives frequency from the scheduler's own utilisation signal, ondemand and conservative are the older sampling governors (aggressive and gradual respectively), and userspace does nothing but obey scaling_setspeed.

The trap is that powersave means two different things. Under active intel_pstate it is the normal, sensible choice: the hardware still boosts under load, and "powersave" only describes the hint. Under acpi-cpufreq or intel_cpufreq it is a hard floor, and you get the same symptom as the userspace bug.

The firmware is the ceiling nothing in Linux can raise. Dell's System Profile setting (Performance, Performance per Watt (OS), Dense Configuration) can cap the range the OS ever sees. cpuinfo_max_freq is where that shows up: if it reads lower than the part's rated turbo, argue with the BIOS, not the kernel.

Which governor for which role

From a small mixed fleet, the rules that have held up:

  • Router, firewall, anything latency-sensitive: performance. A two-core Celeron G4900 routing four networks has nothing to gain from scaling and everything to lose from the ramp-up delay on the first packet of an idle flow. Consistency beats a few watts.
  • File and backup servers with bursty load: schedutil, or ondemand if you are on a kernel old enough to lack it. These machines idle most of the day and then want every megahertz for a scrub or a world rebuild. schedutil reacts on the scheduler's timescale rather than a sampling interval, so the climb is effectively immediate.
  • Hypervisors on modern Xeons: leave intel_pstate in active mode with powersave and stop fiddling. On a Xeon Gold 5222 the hardware's own P-state selection is better informed than anything the kernel can infer through a virtualisation layer.
  • userspace: only for cpupower frequency-set -f, thermal experiments, or benchmarking at a fixed clock. Never as a boot default.

Where the default actually comes from

Three places, in the order they take effect.

The kernel config sets the boot default through CONFIG_CPU_FREQ_DEFAULT_GOV_*. Only one is set, and it must name a governor built in (=y), not modular: a governor compiled as a module cannot be the boot default. On a Gentoo box with CONFIG_IKCONFIG_PROC you can read the running kernel's answer directly: 

zcat /proc/config.gz | grep -E '^#?\s*CONFIG_CPU_FREQ_(DEFAULT_)?GOV'
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y

Userland then gets a say: sys-power/cpupower or cpufrequtils as an init service, an /etc/local.d/*.start script on OpenRC, or tuned and power-profiles-daemon on systemd distributions. Whatever runs last wins.

That matters, because a wrong kernel default does not require a rebuild. On the affected server the complete fix was a local.d one-liner, applied at the next boot and immediately by hand:

for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
  echo schedutil > "$g"
done

Correct the CONFIG_CPU_FREQ_DEFAULT_GOV_* symbol at the next kernel build you were going to do anyway. Do not schedule a reboot for this.

Checking a fleet, and spotting the bug

One line per host tells you nearly everything:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_{driver,governor}
grep MHz /proc/cpuinfo | sort | uniq -c
intel_cpufreq
schedutil
      4 cpu MHz         : 1200.000
      8 cpu MHz         : 2294.000

The tell is not a low reading. It is every core reading exactly cpuinfo_min_freq, to the decimal, on a machine that is demonstrably busy. Idle cores at the floor are normal and correct. Twelve cores pegged at the floor during a compile means the governor is not scaling at all, and the two usual causes are userspace with nothing driving it and powersave under a non-intel_pstate driver.

Confirm the fix by watching the same file while something loads the box. Start a kernel build or a scrub, then:

watch -n1 'grep MHz /proc/cpuinfo | sort -u'

Busy cores should sit near cpuinfo_max_freq and idle ones near cpuinfo_min_freq, with the readings moving as work lands on different cores. Both bounds are in the same directory: 

cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_{min,max}_freq
1200000
2500000

If the busy figure never approaches the second number, you have either a governor that is not scaling or a firmware profile that is holding the ceiling down. The sysfs files tell you which, and they were telling you the whole time.