Unattended Gentoo updates with a boot-chain hold

6 min read

Running emerge -uDN @world from a timer is fine for four hundred packages and dangerous for three. The three are the boot chain: sys-fs/zfs, sys-kernel/gentoo-sources and sys-boot/grub. Each one has an upgrade that is only half done when Portage says it is done, and the other half is a kernel, module or bootloader rebuild that nobody is awake to run.

  • sys-fs/zfs now ships the kernel module alongside the userland. Bump the userland without a matching module for the running kernel and the pool does not import on the next boot.
  • sys-kernel/gentoo-sources installs into a new slot. /usr/src/linux moves, and the next out-of-tree module build targets a tree you never configured or compiled.
  • sys-boot/grub upgrades the files under /boot/grub but does not re-run grub-install. On a machine whose boot code lives on a USB key or a separate ESP, the installed modules and the stage-1 code drift apart quietly and you find out at the next power cycle.

Everything else in @world can flow unattended. The fix is not to stop updating, it is to hold those three and let the rest move.

Pin what is installed, not a version you typed

The failure mode of hand-written masks is that they go stale: you type a version, the tree moves, and six months later the mask is either meaningless or blocking something you wanted. Generate the pins from what is actually merged instead.

qlist -ICve sys-boot/grub sys-kernel/gentoo-sources sys-fs/zfs | sort -V \
  | awk -F'-[0-9]' '{k=$1} k!=p{if(l)print l; p=k} {l=$0} END{print l}' \
  | sed 's/^/>/' >> /etc/portage/package.mask/pinned

On a box here that produces lines of this shape:

>sys-boot/grub-2.12
>sys-fs/zfs-2.4.4
>sys-kernel/gentoo-sources-6.12.41

Each piece earns its place:

  • qlist -I lists installed packages, -v prints full versions, -C drops colour so the pipe stays clean.
  • -e is the one people leave out, and it is the important one. Plain qlist sys-fs/zfs matches substrings, so you also catch zfs-auto-snapshot, and sys-boot/grub drags in grub-themes. Exact matching restricts the output to the package names you asked for.
  • sort -V version-orders the list, so multiple installed slots of gentoo-sources come out oldest to newest.
  • The awk splits each line at the first -<digit>, which is the boundary between the package name and its version, and keeps only the last line seen per name. That is the newest installed version of each package.
  • sed prefixes >, and that is the whole trick.

>sys-fs/zfs-2.4.4 masks everything strictly newer than 2.4.4. It does not mask 2.4.4 itself, so the package can still be re-emerged after a USE flag change, a --changed-deps pull or an emerge @module-rebuild. >= would forbid that too, and you would discover it in the middle of a rebuild you needed.

Leave the command in the file so future-you regenerates the pins instead of editing version numbers by hand:

# Boot chain hold. Regenerate after a controlled bump, do not hand-edit:
#   qlist -ICve sys-boot/grub sys-kernel/gentoo-sources sys-fs/zfs | sort -V \
#     | awk -F'-[0-9]' '{k=$1} k!=p{if(l)print l; p=k} {l=$0} END{print l}' \
#     | sed 's/^/>/' >> /etc/portage/package.mask/pinned

package.mask as a directory

/etc/portage/package.mask can be a plain file or a directory of files, and the directory form is worth the thirty seconds it takes to convert. One file per concern:

/etc/portage/package.mask/pinned    generated boot-chain holds
/etc/portage/package.mask/broken    versions that failed here, with a bug link
/etc/portage/package.mask/policy    things this host will never take

Regenerating the pins then means truncating one file, not surgery on a mixed one. Portage reads every file in the directory.

The mask is deliberately loud. When an update is held back, emerge --pretend -uDN @world says so:

!!! All ebuilds that could satisfy "sys-fs/zfs" have been masked.
!!! (dependency required by "@selected" [set])
  (and: masked by: package.mask)

That is the behaviour you want from an unattended job: a held update reports itself every run instead of silently not happening.

The unattended path

A cron entry or timer that syncs, updates and cleans:

emerge --sync --quiet && \
emerge -uDN --quiet @world && \
emerge --depclean --quiet

Send the output somewhere you read, syslog or your alerting pipe, and keep --depclean at the end rather than folded in, so a failed update does not hand a half-resolved world file to the cleaner.

One important limit, because it is easy to assume otherwise from everything above: package.mask does not protect anything from --depclean. A mask blocks newer versions from being pulled in. It says nothing about whether an installed package is still needed, which is the only question depclean asks. If sys-kernel/gentoo-sources stops being a dependency of anything and is not in your world file, an unattended --depclean will happily unmerge it, pins and all, and you find out at the next kernel build.

If you are going to run depclean unattended, protect the boot chain the way depclean actually respects: keep those packages in @world explicitly (emerge --noreplace sys-kernel/gentoo-sources sys-fs/zfs sys-boot/grub), and consider --depclean --pretend on a schedule with the real run left attended. Unattended and non-interactive is exactly the combination where an unmerge you did not expect is discovered after the reboot rather than before it. The three pins hold the boot chain still while the rest of the system keeps current.

What not to mask: sys-apps/portage itself, sys-libs/glibc, sys-devel/gcc. Pin the toolchain and the unattended path rots into a system that cannot build the updates it is being offered, which is a worse problem than the one you were avoiding. If a specific toolchain version is broken for you, mask that version with a comment in broken, not the package.

The controlled bump

When you do sit down to move the boot chain, it is one attended session:

  1. Delete the lines in /etc/portage/package.mask/pinned.
  2. emerge -uDN @world. The three move together.
  3. Rebuild in order: kernel, then the ZFS module against it, then the initramfs, then grub-install and grub-mkconfig -o /boot/grub/grub.cfg. The order matters because each step consumes the previous one's output.
  4. Reboot and verify before you walk away. zfs --version prints a userland line and a zfs-kmod line, and they must show the same version. findmnt /etc /var confirms the datasets the initramfs was supposed to mount really are mounted. Check that your microcode is loading if you ship it in the initramfs.
  5. Re-run the one-liner. The pins come back at the new versions, and the timer is safe again.

The verification step is the point of the whole exercise. Holding the boot chain buys you a window in which somebody is watching when it moves, and a window you do not use is the same as no hold at all. The userland versus kmod check and the initramfs mount test are covered in more detail in Extract initramfs data and Cleaning Gentoo kernels.

Neighbouring mechanisms

  • --exclude, or -X, skips an atom for a single emerge invocation. Good for a one-off, useless for a scheduled job because nothing records why.
  • package.accept_keywords is the opposite lever: it lets a package go further than the default, where package.mask stops it short. They compose, and a package can be keyworded into ~amd64 and still pinned.
  • package.provided tells Portage a package is installed when it is not. Do not use it here. It suppresses the dependency resolution that would otherwise catch your module and userland drifting apart.
  • --changed-deps will pull rebuilds of packages whose dependencies changed even when versions did not. That is exactly the case > was chosen to allow, and it keeps working under the pins rather than fighting them.