Setting Up a New Hetzner Server

April 27, 2026

256GiB ram, 2x 4TiB drives

Logging in

Don't save the host key for rescue systems

alias ssh.forget='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
ssh.forget root@HOST

Removing RAID Associations

If you have an installation that has previously been used, you may want/need to wipe the partition table. Start with removing raid devices.

cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb2[0] sda2[1]
      2097088 blocks [2/2] [UU]

md1 : active raid1 sdb1[0] sda1[1]
      524224 blocks [2/2] [UU]

unused devices: <none>

Here you need to stop and then remove each array

# all
for i in /dev/md?; do echo $i; mdadm --stop $i 2>/dev/null; mdadm --remove $i 2>/dev/null;done

# one by one
mdadm --stop /dev/md1
mdadm --remove /dev/md1
mdadm --stop /dev/md2
mdadm --remove /dev/md2

Then you can remove the mbr and partition table with dd.

dd if=/dev/zero of=/dev/sda bs=512 count=1

Setting up screen (optional at Hetzner)

Start by getting screen set up.

echo 'caption always "%{= kw}%-w%{= BW}%n %t%{-}%+w %-= %{y}@%H %{r}%1`%{w}| %{g}%l %{w} | %{y}%m/%d/%Y %c %{w}"' >> ~/.screenc
screen -DR

Check to see if UEFI is configured on the the server

efibootmgr
# EFI variables are not supported on this system. => not supported
ls /sys/class/firmware/efi/
# No such file or directory => not supported

Partitioning is harder than it once was. Here is a way to think about it:

UEFI: does your bios support it?
- yes: are you booted with a media that supports EFI? ls /sys/class/firmware/efi
  - yes: use fdisk and partition with GPT/EFI (fat32 /boot is sufficient)
  - no: jump to MBR
- no: jump to MBR
MBR: is your boot disk >2TiB?
- yes: use GPT
  - Is your /boot on raid/lvm?
    - yes: create a bios_boot partition to hold additional grub space, parition with MBR/GPT/parted
    - no: MBR/GPT/parted
- no: use MBR/DOS label, fdisk for simplicity

https://www.gnu.org/software/parted/manual/html_node/set.html

parted --script -a optimal -- /dev/sda \
  mktable gpt \
  mkpart bios_boot  1MiB 2MiB \
  mkpart boot 2MiB 514MiB \
  mkpart swap  514MiB 4GiB \
  mkpart zfs 4GiB -1MiB \
  set 1 bios_grub on \
  set 2 raid on \
  set 3 raid on \
  disk_set pmbr_boot on

Copy the partition table to the other driver

sgdisk /dev/sda -R /dev/sdb
sgdisk -G /dev/sdb

Then you can ensure that the disks are ready

for i in /dev/sd?; do parted $i print; done
Model: ATA ST4000NM0024-1HT (scsi)
Disk /dev/sda: 4001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: pmbr_boot

Number  Start   End     Size    File system  Name       Flags
 1      1049kB  2097kB  1049kB               bios_boot  bios_grub
 2      2097kB  539MB   537MB                boot       raid
 3      539MB   4295MB  3756MB               swap       raid
 4      4295MB  4001GB  3996GB               zfs

Model: ATA ST4000NM0245-1Z2 (scsi)
Disk /dev/sdb: 4001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: pmbr_boot

Number  Start   End     Size    File system  Name       Flags
 1      1049kB  2097kB  1049kB               bios_boot  bios_grub
 2      2097kB  539MB   537MB                boot       raid
 3      539MB   4295MB  3756MB               swap       raid
 4      4295MB  4001GB  3996GB               zfs

RAID

Start with setting up RAID

mdadm --create --level=1 --raid-devices=2 --metadata=0.90 /dev/md2 /dev/sda2 /dev/sdb2
mdadm: array /dev/md2 started.
mdadm --create --level=1 --raid-devices=2 --metadata=0.90 /dev/md3 /dev/sda3 /dev/sdb3
mdadm: array /dev/md3 started.

Ensure the arrays are up and synced

cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb2[1] sda2[0]
      3931072 blocks [2/2] [UU]

md1 : active raid1 sdb1[1] sda1[0]
      262080 blocks [2/2] [UU]

So /dev/md2 is going to be /boot (in ext2) and /dev/md3 will be swap space. Let's start by formating those

mkfs.ext2 -m 0 -L BOOT /dev/md2
mkswap -L SWAP /dev/md3
swapon -L SWAP
mkdir -p /mnt/temp /mnt/gentoo
mount -L BOOT /mnt/temp

LUKS

Create a randown /mnt/temp/loop.crypt and then open it.

cryptsetup luksOpen /mnt/temp/loop.crypt key && echo " * key decrypted"
Enter passphrase for /mnt/temp/loop.crypt:
 * key decrypted

Make sure your key is there.

ls -alh /dev/mapper/key
lrwxrwxrwx 1 root root 7 Nov 10 06:40 /dev/mapper/key -> ../dm-0

Benchmark a couple of algorithms to see where you are comfortable with performance.

cryptsetup -c aes-xts-plain64 -s 256 benchmark
# Tests are approximate using memory only (no storage IO).
# Algorithm |       Key |      Encryption |      Decryption
    aes-xts        256b      2487.2 MiB/s      2534.3 MiB/s

cryptsetup -c aes-xts-plain64 -s 512 benchmark
# Tests are approximate using memory only (no storage IO).
# Algorithm |       Key |      Encryption |      Decryption
    aes-xts        512b      1883.9 MiB/s      1991.7 MiB/s

For me, that's 512. let's encrypt our partitions

cryptsetup luksFormat -c aes-xts-plain64 -s 512  --key-file=/dev/mapper/key /dev/sda4

WARNING!
========
This will overwrite data on /dev/sda2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES

cryptsetup luksFormat -c aes-xts-plain64 -s 512  --key-file=/dev/mapper/key /dev/sdb4

WARNING!
========
This will overwrite data on /dev/sda2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES

We're going to need the UUIDs of those partitions.

blkid /dev/sd?4
/dev/sda3: UUID="548fe34c-3fce-447f-b098-79f86547cb91" TYPE="crypto_LUKS" PARTUUID="5c629cca-e948-f343-afde-0ec9fb44c7ae"
/dev/sdb3: UUID="d0c3a678-d30c-468e-93be-2a4817a91cd1" TYPE="crypto_LUKS" PARTUUID="7ab54734-9943-b749-9be1-0b5d62ea81e4"

Next, we'll need lshw to get the serial numbers off the drives.

apt-get install lshw
lshw  -class disk | grep --color -A 5 -B 5 'serial:\|logical name:'
  *-disk:0
       logical name: /dev/sda
       serial: ZC112A7D
  *-disk:1
       logical name: /dev/sdb
       serial: Z4F0GS2M

Now put the UUID and serial number for each drive together in the following command (repeated once for each partition).

cryptsetup luksOpen UUID="5087f478-1352-43f3-964f-4f91eca80391" --key-file /dev/mapper/key sn-Z4F0GS2M
cryptsetup luksOpen UUID="32f4ad25-ef38-4b35-8b5c-0fe8fadff292" --key-file /dev/mapper/key sn-ZC112A7D

Make sure our partitions are now open and available.

ls -1 /dev/mapper/
control
key
sn-Z4F0GS2M
sn-ZC112A7D

Great, we have two sn-* entries. That's exactly what we want. In the future, if zfs ever complains about a disk, you'll immediately have the serial number to request a replacement. We only want to keep the disk encryption key open while we open the drives. It's time to close that key.

cryptsetup luksClose /dev/mapper/key

Now that luks is set up, we'll put ZFS on top.

ZFS

Hetzner will install zfs for you in the rescue environment now. Thanks!

/usr/local/sbin/zpool
[answer yes to the license]

create zpool

zpool create -f -o ashift=12 -o cachefile= -O atime=off -O relatime=on -O compression=lz4 -O xattr=sa -O mountpoint=none tank mirror /dev/mapper/sn-Z4F0GS2M /dev/mapper/sn-ZC112A7D

now create a space for the root filesystem and mount it

zfs create -o mountpoint=none tank/SYSTEM
zfs create -o mountpoint=legacy tank/SYSTEM/root
mount -t zfs tank/SYSTEM/root /mnt/gentoo/

Create a boot directory and remount the boot device

mkdir /mnt/gentoo/boot
umount /mnt/temp
mount -L BOOT /mnt/gentoo/boot/

Installing Stage

You can follow along in the handbook now.

cd /mnt/gentoo

Then head here to download the current openrc stage 3 install and wget that.

wget https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/20211121T170545Z/stage3-amd64-openrc-XXXXXXX.tar.xz

Now you can extract those files.

tar xpvf stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner

Then updated make.conf

nano -w /mnt/gentoo/etc/portage/make.conf
# ...
USE="-alsa -ipv6 -gif -gtk -gtk2 -jpeg -mp3 -png -tiff -X"

THREADS=9
MAKEOPTS="-j$THREADS"
CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3"
PORTAGE_NICENESS="19"
FEATURES="${FEATURES} parallel-fetch"
EMERGE_DEFAULT_OPTS="$MAKEOPTS --load-average=$THREADS"

ACCEPT_LICENSE="* -@EULA"

Installing the Gentoo base system

Here

mkdir --parents /mnt/gentoo/etc/portage/repos.conf
cp /mnt/gentoo/usr/share/portage/config/repos.conf /mnt/gentoo/etc/portage/repos.conf/gentoo.conf
cp --dereference /etc/resolv.conf /mnt/gentoo/etc/
mount --types proc /proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --make-rslave /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
mount --make-rslave /mnt/gentoo/dev
mount --bind /run /mnt/gentoo/run
mount --make-slave /mnt/gentoo/run
chroot /mnt/gentoo /bin/bash

inside the chroot

source /etc/profile
export PS1="(chroot) ${PS1}"

continuing the installation

emerge-webrsync
eselect news read all
eselect news purge
eselect profile set 1
echo "America/Chicago" > /etc/timezone
emerge --config sys-libs/timezone-data
locale-gen
env-update && source /etc/profile && export PS1="(chroot) ${PS1}"

Kernel

Install kernel sources

emerge --ask sys-kernel/gentoo-sources
eselect kernel set 1
pushd /usr/src/linux
make menuconfig
make -j9 && make -j9 modules_install && make -j9 install && make modules_prepare
popd

Determine what modules are loaded from the livecd.

lspci -k | grep -i Kernel | sed 's/.*: //' | sort | uniq
ahci
ehci-pci
hswep_uncore
i2c_i801
i801_smbus
igb
lpc_ich
mei_me
pcieport
xhci_hcd

Below are examples of what you need to configure.

General setup  --->
  Local version - append to kernel release
  [*] Automatically append version information to the version string
Enable the block layer --->
  Partition Types --->
    [*] Configuring the System partition selection
    [*] EFI GUID Partition support
Device Drivers  --->
  Multiple devices driver support (RAID and LVM)  --->
    RAID support
      <*> RAID-0 (striping) mode
      <*> RAID-1 (mirroring) mode
    Device mapper support
      <*> Crypt target support
  Network device support  --->
    Ethernet driver support (NEW)  --->
      <*> Intel(R) 82575/82576 PCI-Express Gigabit Ethernet support
File systems  --->
  <*> Second extended fs support
Cryptographic API  --->
  <*>   AES cipher algorithms (AES-NI)
  <*>   LZO compression algorithm
  <*>   LZ4 compression algorithm

Configuring the System

asdf

emerge --ask --noreplace net-misc/netifrc
nano -w /etc/conf.d/hostname /etc/conf.d/net
config_eth0="203.0.113.252 netmask 255.255.255.192 brd 0.0.0.0"
routes_eth0="default via 203.0.113.193"

pushd /etc/init.d && ln -s net.lo net.eth0 && rc-update add net.eth0 default && rc-update add sshd default && popd
rm -rf /etc/portage/package.use && touch /etc/portage/package.use
echo "sys-fs/mdadm static" >> /etc/portage/package.use
emerge -av sys-fs/zfs sys-fs/e2fsprogs sys-fs/mdadm sys-boot/grub dev-vcs/git htop

grub default

nano -w /etc/default/grub
# Default menu entry
GRUB_DEFAULT=saved

# Boot the default entry this many seconds after the menu is displayed
GRUB_TIMEOUT=3

GRUB_CMDLINE_LINUX="net.ifnames=0 sshd sshd_wait=60 sshd_port=2222 binit_net_if=eth0 binit_net_addr=203.0.113.252/26 binit_net_gw=203.0.113.193 root=/dev/mapper/FAKE quiet elevator=noop"

Now try to install grub. We need to install it to both disks to allow for failure of a single disk on boot. Make sure to address any errors before proceeding.

grub-install /dev/sda
grub-install /dev/sdb

Initramfs

this is init

pushd /usr/src/
git clone --depth=1 https://bitbucket.org/piotrkarbowski/better-initramfs.git
pushd better-initramfs
make bootstrap-all

copy a couple of files

mkdir sourceroot/root
mv /boot/loop.crypt sourceroot/root/

sourceroot/root/unlock.sh

#!/bin/sh
#
# Hetzner S7 unlock script
#

set -e

KEY_HOLDER="/root/loop.crypt"
NEWROOT="/newroot"

cryptsetup luksOpen "$KEY_HOLDER" key && echo " * key decrypted"

# check to ensure key is a block device now
if [ ! -b /dev/mapper/key ]
then
  echo "decryption failed"
  exit 2
fi

echo " * about to decrypt zfs devices"
cryptsetup luksOpen UUID="5087f478-1352-43f3-964f-4f91eca80391" --key-file /dev/mapper/key sn-Z4F0GS2M
cryptsetup luksOpen UUID="32f4ad25-ef38-4b35-8b5c-0fe8fadff292" --key-file /dev/mapper/key sn-ZC112A7D

# close key
echo " * closing key"
cryptsetup luksClose /dev/mapper/key

echo " * loading zfs module"
/sbin/modprobe zfs

echo " * importing tank"
zpool import -f tank -R "$NEWROOT"

echo " * mounting datasets"

mount -t zfs -o zfsutil tank/SYSTEM/root "$NEWROOT"

echo " * resuming boot in 3 seconds, you will be disconnected"
sleep 3s
resume-boot

make it executable

chmod 755 sourceroot/root/unlock.sh

Now copy keys to sourceroot/authorized_keys and also to /root/.ssh/authorized_keys

cat >> sourceroot/authorized_keys
mkdir /root/.ssh
cp sourceroot/authorized_keys /root/.ssh

Create initramfs

export ZPOOL_VDEV_NAME_PATH=YES
THREADS=$(grep -c ^processor /proc/cpuinfo)
BETTER="/usr/src/better-initramfs/"
apps="fsck.zfs htop nano reboot zed mount.zfs shutdown zfs zpool"
NAME=$(ls -tr1 /lib/modules | tail -n 1)
rsync -r --delete --progress /lib/modules "$BETTER"/sourceroot/lib/
mkdir "$BETTER"/sourceroot/sbin/ "$BETTER"/sourceroot/lib64/
pushd "$BETTER"/sourceroot/sbin/
rm -rf "$BETTER"/sourceroot/sbin/*
for app in $apps; do
  cp -v $(which $app) .
done
echo " * copying libs (libzfs, etc) for bins"
for i in *; do lddtree -l $(which "$i" 2>/dev/null) 2>/dev/null ;done | grep -v ^/sbin/ | grep lib64 | sort | uniq | while read line ; do cp $line "$BETTER"/sourceroot/lib64/; done
cp /usr/lib/gcc/$(eselect gcc show | sed 's:\(.*\)-:\1/:g')/libgcc_s.so.1 "$BETTER"/sourceroot/lib64/
popd
echo " * ncurses libs"
mkdir -p "$BETTER"/sourceroot/etc/
rm -rf "$BETTER"/sourceroot/etc/terminfo
find /lib64/ -iname "*libncurses*" -exec cp {} "$BETTER"/sourceroot/lib64/ \;
cp -r /etc/terminfo "$BETTER"/sourceroot/etc/
cd "$BETTER"
make prepare
make image
cp -v "$BETTER"/output/initramfs.cpio.gz /boot/initrd-"$NAME"
echo " * making grub boot menu"
grub-mkconfig -o /boot/grub/grub.cfg
echo " * here are the menu entries"
awk -F\' '/menuentry / {print $2}' /boot/grub/grub.cfg

Now, since grub uses "saved" we need to specify the default entry

grub-set-default "Gentoo GNU/Linux, with Linux 5.10.76-gentoo-r1-2021-11-27-1427"

Wrap Up

This is straight from the handbook

exit

cd
umount -l /mnt/gentoo/dev{/shm,/pts,}
umount -R /mnt/gentoo
reboot

Keep your fingers crossed as you reboot. Remember, dropbear should start up and you see that it is pingable. Remember to log in with port 2222

Recovery

When something goes wrong, here is what you can do. First, boot into rescue mode. Next, we need to extract an open our encryption key.

mkdir /mnt/{temp,gentoo} /root/temp
mount -L BOOT /mnt/temp/
cp /mnt/temp/initrd* /root/temp/
umount /mnt/temp
pushd /root/temp/
zcat /boot/initrd-2.6.18-164.6.1.el5.img | cpio -idmv
cd root
cryptsetup luksOpen loop.crypt key

Next, we need to open the encrypted zfs partitions. Looks for the lines in unlock.sh.

# cat unlock.sh and execute the following similar commands
cryptsetup luksOpen UUID="5087f478-1352-43f3-964f-4f91eca80391" --key-file /dev/mapper/key sn-Z4F0GS2M
cryptsetup luksOpen UUID="32f4ad25-ef38-4b35-8b5c-0fe8fadff292" --key-file /dev/mapper/key sn-ZC112A7D

Re-install zfs in the rescue environment.

/usr/local/sbin/zpool # say 'y'
/sbin/modprobe zfs
zpool import -f tank


mount -t zfs tank/SYSTEM/root /mnt/gentoo/
ls /mnt/gentoo/

Now you can chroot like normal.

mount --types proc /proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --make-rslave /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
mount --make-rslave /mnt/gentoo/dev
mount --bind /run /mnt/gentoo/run
mount --make-slave /mnt/gentoo/run
chroot /mnt/gentoo /bin/bash

And refresh your profile.

mount -L BOOT /boot/
source /etc/profile
export PS1="(chroot) ${PS1}"

The actually fixing is up to you. Good luck!

Next Steps

Here is an overview gist of the last server migration.