Upgrading a Fedora 39 Installation on a Btrfs Filesystem with an openSUSE Style Subvolume Layout to Fedora 40
Published: May 18, 2024, 9 p.m.
Updated: Nov. 16, 2024, 6 p.m.
In a recent article on this site, a process to convert a Fedora 39 installation on a Btrfs filesystem with a simple subvolume layout to the hierarchical subvolume layout style of openSUSE with Snapper integration, automatic creation of snapshots, GRUB menu integration of snapshots, and other conveniences was presented. Since then Fedora 40 (and Ultramarine Linux 40) has been released. The installation and configuration presented in the earlier article survived a release upgrade to 40 with the Btrfs/Snapper configuration intact.
This article describes the release upgrade process using the CLI approach (although the native GUI package manager of the desketop environment used in the installation, KDE's Discover, is able to perform the release upgrade). More importantly, it describes the post-upgrade fixes to GRUB necessary, partly, because of Fedora's non-standard implementation of GRUB.
Updates
2024-11-16: Although this article describes the upgrade from Fedora 30 to Fedora 40 with the custom Btrfs/Snapper described in the linked article, I have since used the process described here to upgrade the same installation again from Fedora 40 to Fedora 41.
Introduction
Fedora is a highly refined and reliable distribution. As such, its package management infrastructure provides a reliable release upgrade experience, with either a command line tool, or any of the GUI package management utilities provided by its various editions, including KDE's Discover included with the Plasma Spin. I have used the upgrade tools in past Fedora editions without any issues, but I was not sure that the the upgrade tools would leave the Btrfs/Snapper/grub-btrfs configuration on a Fedora 39 (actually Ultramarine Linux 39) installation on a Btrfs filesystem that was converted to an openSUSE style Btrfs subvolume layout intact.
It did! However, GRUB packages had to be reinstalled and the GRUB firmware application installed to the ESP (EFI System Partition) had to be reinstalled.
Snapper Rollback Before Upgrade
Before upgrading to 40, I performed a rollback in order to make sure that the upgrade would modify a new snapshot subvolume that becomes the root of the filesystem hierarchy and not the initial snapshot subvolume which contained the original installation. This was completely unnecessary ,but I did it anyway to have a new snapshot subvolume containing the root of the filesystem hierarchy for the new version of Fedora. This did result in having to repair the GRUB issue mentioned below (Section Rebooting After Upgrade). It is instructive, however, because problem would occur when rolling back the Fedora system with this Btrfs/Snapper configuration even without a release upgrade and would have to be repaired as described in Section: Post-Upgrade Fix for GRUB, below. The following image, Figure 1, illustrates the initial state of the system before the rollback and the rollback command and its output.
Figure 1: The State of the System Relevant to the Btrfs/Snapper Configuration Before Rollback
Before the rollback the root of the filesystem hierarchy was in Snapshot 1. After the rollback it will eb Snapshot 191.
The Konsole window at the top shows the outputs of three commands:
snapper list
btrfs subvolume get-default /
and
cat /proc/cmdline
that illustrate the state before the rollback. The first lists the existing snapshots available before the rollback. Note, for comparison with the state after the rollback, that the latest snapshot was number 189. The second shows that before the rollback, the default subvolume was
/@-0/.snapshots/1/snapshot
This was the initial snapshot subvolume to which the original filesystem hierarchy root was moved by the process described in A Fedora Installation with an openSUSE Style Btrfs Subvolume Layout and Snapper Integration for System Snapshots and Rollbacks, the 1 indicating Snapshot #1. And the third shows that before the rollback the kernel image path was in the default subvolume.
The middle Konsole window shows the GRUB menu configuration, /boot/grub2/grub.cfg, generated by grub2-mkconfig. This also shows that the kernel image and initial ram disk images are in the Snapshot #1.
In the bottom window, the Snapper rollback command is executed to rollback to Snapshot #189, a snapshot created at the last boot by the systemd timer snapper-boot.timer. The output shows that it created a read-only copy of the current, before rollback snapshot subvolume that is the current root of the filesystem hierarchy (Snapshot 190), and a read-write snapshot copy of the snapshot to which to rollback, Snapshot 191. It sets the default Btrfs subvolume as the new read-write snapshot, which will be the new root of the filesystem hierarchy and mounted automatically at boot by /etc/fstab (see A Fedora Installation with an openSUSE Style Btrfs Subvolume Layout and Snapper Integration for System Snapshots and Rollbacks). After the rollback, on the next boot, the new root of the filesystem hierarchy will be in Snapshot 191, whereas before the rollback it was in Snapshot 1.
When the system was rebooted after the rollback command, the GRUB menu did not reflect the new snapshot subvolume as the new system root. It was necessary to manually edit the default GRUB menu entry by pressing e on the menu item and then changing the kernel and initial RAM disk image paths from /@-0/.snapshots/1/snapshot to /@-0/.snapshots/191/snapshot. The edited GRUB menu entry is depicted in the following image (Figure 2).
Figure 2: Edited GRUB Menu Entry After Rollback
On the next boot after rollback it was necessary to edit the GRUB menu entry by pressing e on the menu item and changing the kernel and initial RAM disk image paths to the new snapshot subvolume that contains the system filesystem hierarchy root.
Then after the desktop was started, it was then necessary to update the GRUB menu configuration with
grub2-mkconfig -o /boot/grub2/grub.cfg
so that the GRUB menu would reflect the new snapshot subvolume as the location of the new filesystem hierarchy root.
The state of the system after rebooting with the manually edited GRUB menu is demonstrated by the next image, Figure 3, which shows a Konsole window, at the left of the screen, with the same three commands as the topmost window in Figure 1.
Figure 3: The State of the System Relevant to the Btrfs/Snapper Configuration After Rollback
After the rollback the root of the filesystem hierarchy was Snapshot 191, whereas before it was Snapshot 1.
The output of the first command,
snapper list
indicates, by the * to the right of 191 that Snapshot 191 is the snapshot currently mounted at / and the snapshot that will be mounted at / at next boot (because it is the default subvolume). The second shows the kernel command line indicating that the kernel image was in Snapshot 191 with subvolume path /@-0/.snapshots/191/snapshot. The third output shows that the new default subvolume is /@-0/.snapshots/191/snapshot.
The Konsole windows at the right of the image (Figure 3) show the GRUB menu configuration /boot/grub2/grub.cfg before the update (top window), the bottom of the output of the grub2-mkconfig command, which incorporates grub-btrfs actions (middle window), and the GRUB menu configuration after the GRUB configuration update (bottom window). After the update, the GRUB menu configuration correctly looks for the kernel and initial RAM disk images at the new system root, in /@-0/.snapshots/191/snapshots, whereas before it had been at /@-0/.snapshots/1/snapshots (top window).
Release Upgrade
After the rollback, making Snapshot 191 the subvolume containing the root of the system filesystem hierarchy, instead of the original Snapshot 1, it was time to perfom the upgrade. The process steps are listed below. The Fedora Quick Docs page referenced below has details, including procedures for resolving errors caused by dependency issues, which were not necessary in my case.
-
Fully update the current system.
sudo dnf upgrade --refresh
-
Download the upgraded packages and prepare the upgrade transaction.
sudo dnf system-upgrade download --releasever=40
It may be necessary to use the --allowerasing if appropriate depending on the errors displayed with the command. -
Start the upgrade. The actual upgrade will be offline after an automatic reboot.
sudo dnf system-upgrade reboot
The following set of images illustrate the two types of user feedback provided by the upgrade tool during the upgrade. The first image shows the screen presented by default during the offline upgrade, indicating the progress graphically. Swithcing to one of the other virtual consoles by pressing Ctrl + Alt + FnX follows the messages created by the upgrade tool.
Figures 4a and 4b: The Offline Update
The first image shows the screen presented by default during the offline upgrade, which indicates the progress. Switching to one of the other virtual consoles by pressing Ctrl + Alt + FnX follows the messages created by the upgrade tool.
Rebooting After Upgrade
The Btrfs/Snapper related aspects of the Fedora 39 system converted to the openSUSE style of subvolume layout with Snapper integration survived the upgrade to Fedora 40. There was a challenge, however, with respect to the non-standard way Fedora uses GRUB, presumably to increase reliability. This non-standard feature is described in the second reference, but briefly, according to the reference, the GRUB configuration normally written to /boot/grub/grub.cfg, or more precisely, in Fedora's case, /boot/grub2/grub.cfg, is actually used to generate the GRUB menu at boot only after being referenced by another GRUB configuration file written to the Fedora directory in the ESP at /boot/efi/EFI/fedora/grub.cfg during the installation of the grub-common package. The merged configuration files are what actually generates the GRUB menu. (In my case, with the integration of grub-btrfs's -- see A Fedora Installation with an openSUSE Style Btrfs Subvolume Layout and Snapper Integration for System Snapshots and Rollbacks -- own configuration into the merged to GRUB menu configuration to produce the GRUB menu item listing available snapshots.)
The following image (Figure 5) shows two Konsole panes showing the contents of the directories which contain the two grub.cfg files. The left pane displays a tree listing of /boot/grub2/ and the right pane displays a tree listing of /boot/efi/EFI/fedora -- Fedora's directory in the EFI System Partition.
Figure 5: An Element of Fedora's Non-Standard Use of GRUB
The GRUB configuration actually used by Fedora's GRUB firmware application is generated by a merger of /boot/grub2/grub.cfg and another stored in the EFI System Partition, accessible in the system from /boot/efi/EFI/fedora. The image shows the contents of /boot/grub2/ and /boot/efi/EFI/fedora.
Comparing the GRUB configuration file /boot/grub2/grub.cfg, which contains the subvolume path from main top-level Btrfs subvolume, illustrated in various images in this article, and the GRUB configuration file in the ESP, /boot/efi/EFI/fedora/grub.cfg, it seems the only thing the latter adds to the final configuration is the value of the $prefix GRUB variable. After the initial of Fedora 39 (actually, Ultramarine 39) as described in A Fedora Installation with an openSUSE Style Btrfs Subvolume Layout and Snapper Integration for System Snapshots and Rollbacks, /boot/efi/EFI/fedora/grub.cfg has the contents:
search --no-floppy --root-dev-only --fs-uuid --set dev 1a773ae9-02a6-4e95-bf07-c500427e652b set prefix=($dev)/@-0/.snapshots/1/snapshot/boot/grub2 export $prefix configfile $prefix/grub.cfg
The first lines sets the dev parameter whose value is the UUID containing the Btrfs partition. The second line sets the prefix variable which is the path to the directory containing the normal GRUB configuration generated by grub2-mkconfig. Note that the path contains the full subvolume path of the subvolume containing the current root of the filesystem hierarchy. Note especially the 1 in the path meaning that is the directory that contains Snapshot #1. The last line references the normal GRUB configuration file causing it to be merged with the stub GRUB configuration file /boot/efi/EFI/fedora/grub.cfg.
The problem after the upgrade, consequently, is that the grub.cfg in the ESP, which only gets updated when the GRUB firmware is reinstalled in the ESP (see next section), does not get updated by the grub2-mkconfig command, and as a result did not reflect the new snapshot subvolume after the rollback. So after the upgrade, even after updating the GRUB configuration after the rollback, the GRUB menu after the release upgrade did not reflect the correct kernel image or initial RAM disk image paths; it used the path /@-0/.snapshots/1/snapshot/boot/vmlinuz-xxx... instead of/@-0/.snapshots/191/snapshot/boot/vmlinuz-xxx.... I simply did not notice the problem because I immediately performed the upgrade after only confirming /boot/grub2/grub.cfg was updated to reflect paths in the new snapshot subvolume, and not the grub.cfg in the ESP. (This issue is corrected in the next section.)
The Btrfs/Snapper Related Characteristics of the Upgraded System
All Btrfs/Snapper relaed aspects of the system survived the upgrade. However, because, Fedora uses a duplicate of /boot/grub2/grub.cfg stored in the ESP, a non-standard use of GRUB, the first boot after upgrade required editing the kernel and initial RAM disk paths before booting into the correct snapshot subvolume.
So, on the first boot after the upgrade, it was necessary to edit the GRUB menu entry manually, as discussed above, to reflect the correct snapshot subvolume in the kernel and initial RAM disk paths in order to boot the system. After booting in this way, I confirmed that the Btrfs/Snapper configurations, and /boot/grub2/grub.cfg reflected the correct snapshot subvolume as containing the filesystem hierarchy root, as shown in the following image.
Figure 6: The Btrfs/Snapper Related Characteristics of the Upgraded System
All Btrfs/Snapper relaed aspects of the system survived the upgrade. However, because, Fedora uses a duplicate of /boot/grub2/grub.cfg stored in the ESP, a non-standard use of GRUB, the first boot after upgrade required editing the kernel and initial RAM disk paths before booting into the correct snapshot subvolume.
The image shows that the default Btrfs subvolume is set to Snapshot 191, /boot/grub2/grub.cfg contains kernel and initial RAM disk paths that include Snapshot 191, and the kernel command line includes Snapshot 191 (because the path was manually edited in the GRUB menu). However, as described above, because the GRUB configuration in Fedora's directory in the ESP, /boot/efi/EFI/fedora/grub.cfg, was not updated it reflected the pre-rollback snapshot subvolume as that containing the root of the filesystem hierarchy, GRUB was not able to start find the kernel or initial RAM disk images.
Post-Upgrade Fix for GRUB
After booting into the Fedora 40 (Ultramarine 40) system by editing the GRUB menu entry for reasons described above, and confirming that the Btrfs/Snapper configurations and /boot/grub2/grub.cfg file reflected the correct snapshot subvolume for the system root, it was necessary to update the duplicate GRUB configuration file in Fedora's directory in the ESP with the following process.
-
Delete the grub.cfg file in the ESP.
sudo rm -f /boot/efi/EFI/fedora/grub.cfg
-
Reinstall the GRUB packages grub2-efi, grub2-efi-modules, and several shim packages which can be specified to DNF as shim-\*. The command to install these, and the first part of its output is shown in the following listing:
brook on Ultramarine-16ITH6 ~ ❯ sudo dnf reinstall grub2-efi grub2-efi-modules shim-\* [sudo] password for brook: Fedora 40 - x86_64 - Updates 2.8 kB/s | 21 kB 00:07 Fedora 40 - x86_64 - Updates 25 kB/s | 2.8 MB 01:54 Terra 40 657 B/s | 1.0 kB 00:01 Terra 40 1.3 kB/s | 19 kB 00:14 Ultramarine Linux 40 628 B/s | 1.0 kB 00:01 Ultramarine Linux 40 2.6 kB/s | 16 kB 00:06 Dependencies resolved. ============================================================================================================ Package Architecture Version Repository Size ============================================================================================================ Reinstalling: grub2-efi-ia32 x86_64 1:2.06-121.fc40 updates 2.1 M grub2-efi-x64 x86_64 1:2.06-121.fc40 updates 2.1 M grub2-efi-x64-modules noarch 1:2.06-121.fc40 updates 1.1 M shim-ia32 x86_64 15.8-3 fedora 413 k shim-unsigned-x64 x86_64 15.8-2 fedora 460 k shim-x64 x86_64 15.8-3 fedora 467 k Transaction Summary ============================================================================================================ Total download size: 6.5 M Installed size: 24 M Is this ok [Y/n]:
-
Reinstall the grub2-common. (Note that the following output is from GRUB fix after the upgrade of Fedora 40 to Fedora 41, not from the upgrade of Fedora 39 to Fedora 40, due to my failure to copy the output when performing the upgrade to 40 from 39. The formatting of the DNF output is different from other DNF output in this article because DNF was updated to DNF 5 in Fedora 41.)
19:08:03 USER: brook HOST: Ultramarine-16ITH6 PCD: 22s ~ ❯$ sudo dnf reinstall grub2-common Updating and loading repositories: Repositories loaded. Package Arch Version Repository Size Reinstalling: grub2-common noarch 1:2.12-10.fc41 fedora 6.1 MiB replacing grub2-common noarch 1:2.12-10.fc41 fedora 6.1 MiB Transaction Summary: Reinstalling: 1 package Replacing: 1 package Total size of inbound packages is 1 MiB. Need to download 1 MiB. After this operation, 0 B extra will be used (install 6 MiB, remove 6 MiB). Is this ok [Y/n]:
-
Uprate the GRUB configuration by executing grub2-mkconfig
100% 19:17:48 USER: brook HOST: Ultramarine-16ITH6 PCD: 9m42s ~ ❯$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg [sudo] password for brook: Generating grub configuration file ... Found linux image: /boot/vmlinuz-6.11.7-300.fc41.x86_64 Found initrd image: /boot/initramfs-6.11.7-300.fc41.x86_64.img Found linux image: /boot/vmlinuz-6.11.6-200.fc40.x86_64 Found initrd image: /boot/initramfs-6.11.6-200.fc40.x86_64.img Found linux image: /boot/vmlinuz-6.11.5-200.fc40.x86_64 Found initrd image: /boot/initramfs-6.11.5-200.fc40.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-a6c93ae8fb4347699af7cd22cd207c55 Found initrd image: /boot/initramfs-0-rescue-a6c93ae8fb4347699af7cd22cd207c55.img Found Windows Boot Manager on /dev/nvme1n1p1@/EFI/Microsoft/Boot/bootmgfw.efi Found openSUSE Tumbleweed on /dev/nvme1n1p5 Found Arch Linux on /dev/nvme1n1p6 Found Ubuntu 24.10 on /dev/nvme1n1p7 Found Ubuntu 24.10 on /dev/nvme1n1p7 Found Ultramarine Linux 39 (KDE Plasma Edition) on /dev/nvme1n1p8 Found Garuda Linux on /dev/nvme1n1p9 Adding boot menu entry for UEFI Firmware Settings ... Detecting snapshots ... Found snapshot: 2024-11-16 19:00:10 | @-0/.snapshots/817/snapshot | single | timeline | Found snapshot: 2024-11-16 18:19:20 | @-0/.snapshots/816/snapshot | single | timeline ... excised ... Found snapshot: 2024-10-01 19:11:57 | @-0/.snapshots/740/snapshot | post | /bin/dnf -y install --nogpgcheck --disablerepo=* /tmp/akmods.5o0UqH3h/results/kmod-VirtualBox-6.10.11-200.fc40.x86_64-7.1.0-1.um40.x86_64.rpm | Found snapshot: 2024-10-01 19:11:50 | @-0/.snapshots/739/snapshot | pre | /bin/dnf -y install --nogpgcheck --disablerepo=* /tmp/akmods.5o0UqH3h/results/kmod-VirtualBox-6.10.11-200.fc40.x86_64-7.1.0-1.um40.x86_64.rpm | Found snapshot: 2024-05-16 19:49:49 | @-0/.snapshots/190/snapshot | single | Rollback to demo rollback and change soot snapshot subvolume before Upgrade to 40. | Found snapshot: 2024-03-26 18:00:00 | @-0/.snapshots/13/snapshot | single | timeline | Found 43 snapshot(s) Unmount /tmp/grub-btrfs.T7BIrt0R6f .. Success done
The next image shows the difference in the system before the above process to fix the post upgrade GRUB issue was fixed and after it was fixed. The Kpnsole windows in the left half of the screen reflect the before condition and those on the right, the after condition. The topmost windows show the contents of /boot/efi/EFI/fedora/grub.cfg before and after the fix. Before the fix, the paths reflected the pre-upgrade, pre-rollback snapshot subvolume (Snapshot #1) and after the fix they correctly reflected the correct post-rollback snapshot subvolume.
Figure 7: GRUB Configuration Before and After the Post-Upgrade GRUB Fix
It is necessary to reinstall GRUB packages, including grub2-common to update Fedora's GRUB configuration file in the ESP, /boot/efi/EFI/fedora/grub.cfg
Conclusion
Having upgraded a Fedora installation with a customized Btrfs subvolume layout with Snapper integration in the openSUSE style from release 39 to release 40, and again from 40 to 41, it is clear that the openSUSE style configuration is reliably retained after release upgrades. If the system is not rolled back before upgrade, as I did for superficial reasons, there would be no issues with GRUB described in the article, and the upgrade would complete easily and successfully.
However, if performing a rollback before the upgrade, a repair to the GRUB configuration is necessary due to the non-standard stub GRUB configuration file placed in the ESP by Fedora. This GRUB issue is not related to the upgrade, but something that would have to be confronted anytime a rollback is performed because of the non-standard GRUB configuration in the ESP.