osbuild: use bootc install to deploy the container #4224

jbtrystram · 2025-07-17T15:21:55Z

Instead of deploying the container to the tree then copy all the contents to the disk image, use bootc to directly manage the installation to the target filesystems.

Right now this requires to use the image as the buildroot so this requires python (for osbuild). This is tracked in [1].

[1] bootc-dev/bootc#1410 Requires osbuild/osbuild#2149

openshift-ci · 2025-07-17T15:21:59Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

gemini-code-assist

Code Review

The pull request introduces changes to use bootc install to deploy the container, which simplifies the image build process. There are a few critical issues in the YAML manifest related to copy-paste errors that lead to incorrect configurations for the 4k image builds and missing options for loopback devices. These issues need to be addressed.

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

build.sh

dustymabe · 2025-07-17T20:35:37Z

I switched the CI on this to run against rawhide (contains python) so we could actually test the change.

dustymabe · 2025-07-19T21:31:47Z

A few diffs picked up by cosa diff --metal from #4226

cosa-diff-metal.txt

We should probably profile each diff (maybe in coreos/fedora-coreos-tracker#1827) and evaluate whether it's a change we want to make or not.

dustymabe · 2025-07-19T21:43:24Z

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

jbtrystram · 2025-07-21T13:22:50Z

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

do you mind sharing more logs ? What I am getting locally is ignition failing on coreos/fedora-coreos-tracker#1250

dustymabe · 2025-07-21T13:39:27Z

Ahh. I see that too now:

[    4.726843] ignition[875]: Ignition failed: failed to create users/groups: failed to configure users: failed to create user "core": exit status 10: Cmd: "useradd" "--root" "/sysroot" "--create-home" "--password" "*" "--comment" "CoreOS Admin" "--groups" "adm,sudo,systemd-journal,wheel" "core" Stdout: "" Stderr: "useradd: cannot lock /etc/group; try again later.\n"

jbtrystram · 2025-07-21T15:31:31Z

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

looks like removing those make the boot process go further (ignition completes), and out of the initramfs but fail to mount the boot partition.

jbtrystram · 2025-07-24T15:19:53Z

Blocked on bootc-dev/bootc#1441

jbtrystram · 2025-07-28T16:52:09Z

ok this works with the following PRs :

for the bootc PR, it can be built then added into the image through overrides/rootfs. Make sure to build rawhide.

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

jbtrystram · 2025-07-29T12:44:45Z

follow-up : either find a way to get the boot components inside cosa, or change the bootc code to call bootupd from the deployed root . I think the latter is preferable.
I filed bootc-dev/bootc#1455

jbtrystram · 2025-07-29T20:29:22Z

follow-up : either find a way to get the boot components inside cosa, or change the bootc code to call bootupd from the deployed root . I think the latter is preferable. I filed bootc-dev/bootc#1455

Made bootc-dev/bootc#1460
With this, we no longer require to use the container as the buildroot, cosa works, so we could do that on all streams.

Instead of deploying the container to the tree then copy all the contents to the disk image, use bootc to directly manage the installation to the target filesystems. Right now this requires to use the image as the buildroot so this requires python (for osbuild). This is tracked in [1]. [1] bootc-dev/bootc#1410 Requires osbuild/osbuild#2149

Bootc is looking for the prepare-root config file in the buildroot environnement because the main assumption is that it's run from the target container. However, in osbuild, it's run from te buildroot, because podman inside bwrap (inside supermin in our case) causes issues. It's fine for RHCOS and SCOS where we use the target container as the buildroot but we cannot do that for FCOS because we require python in the buildroot. For now, insert a prepare-root file in the supermin VM (use as the buildroot for osbuild) until either : - bootc learn to look into the container for it [1] - we ship python in our images and can use them as buildroot. Another approach would be to layer python and the osbuild dependencies on top of our image and use that as the buildroot, but that would create room for packages drift (what was in the repos at build time?). At least using COSA it's easier to keep track of versions. [1] bootc-dev/bootc#1410

Theses are patch from the following PRs: osbuild/osbuild#2152 osbuild/osbuild#2149

jbtrystram · 2025-07-30T07:44:35Z

Alright, marking this as ready for review as all the bits are in place.
I guess i need to update the osbuild manifest or the other arches as well, but I'll do that after a review to reduce the amount of back and forth.

This will need a release of bootc.

openshift-ci · 2025-07-30T08:17:09Z

@jbtrystram: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/rhcos	`310bd60`	link	true	`/test rhcos`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

dustymabe

Some comments.

I think there are a few things we need to iron out before we can really move forward with this:

supporting both old and new paths at the same time

Do we need to? Usually when we make a change this large we roll it out slowly, which means we have to support both ways for some time.

This PR is ignoring that fact, but TBH looking at OSBuild configs that support both would be pretty intimidating, so I'm not excited about trying to do that either. I'd be interested in @jlebon or @travier's thoughts.

We need to make sure any/every diff that exists between images generated this way and the old way are considered and acknowleged as acceptable before we'd make this change.

dustymabe · 2025-07-31T16:32:12Z

src/cmd-osbuild

@@ -397,6 +397,10 @@ main() {
    fi

    outdir=$(mktemp -p "${tmp_builddir}" -d)
+    # To get a shell in the osbuild supervin VM uncomment this.


Suggested change

# To get a shell in the osbuild supervin VM uncomment this.

# To get a shell in the osbuild supermin VM uncomment this.

dustymabe · 2025-07-31T16:38:11Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

+          boot-mount-spec: ""
+          root-mount-spec: ""


Might be worth adding a comment above these two and maybe even linking to bootc-dev/bootc#1441

dustymabe · 2025-07-31T16:41:35Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

+            - mpp-format-string: '{extra_kargs}'
+          target-imgref:
+            mpp-format-string: '{container_imgref}'
+          stateroot: fedora-coreos


We'd need this to work for RHCOS and FCOS:

Suggested change

stateroot: fedora-coreos

stateroot:

mpp-format-string: '{osname}'

dustymabe · 2025-07-31T16:43:35Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

-      - type: org.osbuild.bootupd
+      # set up the `ignition.firstboot` stamp at the end because
+      # bootc want empty filesystems
+      - type: org.osbuild.ignition
        options:
-          static-configs: true
-          deployment:
-            default: true


Does bootc install to filesystem run bootupd with all the same settings as we were doing in the past?

bootc :

> bootupctl backend install --write-uuid --src-root /run/osbuild/mounts/ostree/deploy/fedora-coreos/deploy/bcbf1077fcd3a702c763474103ef1d160d948e84168a966432fdce48010be75b.0 --device /dev/loop0 /run/osbuild/mounts

Our pipeline :

/usr/bin/bootupctl backend install --device /dev/loop0 --with-static-configs /run/osbuild/mounts

So yeah. Looking at the code I see --write-uuid implies --with-static-configs
The --src-root is replaced by a chroot in the org.osbuild.bootupd stage but they are functionnaly equivalent.

One thing i am not sure of is writing the disks uuid . In our case we change them during the first boot so I need to investigate if that creates problems.

Thanks for looking into it!

dustymabe · 2025-07-31T16:45:24Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

+            - mpp-format-string: '{extra_kargs}'
+          target-imgref:
+            mpp-format-string: '{container_imgref}'
+          stateroot: fedora-coreos


same as https://github.com/coreos/coreos-assembler/pull/4224/files#r2245883530

dustymabe · 2025-07-31T16:47:00Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

-            source: disk
-            partition:
-              mpp-format-int: '{image4k.layout[''root''].partnum}'
+              mpp-format-int: '{image.layout[''root''].partnum}'


Suggested change

mpp-format-int: '{image.layout[''root''].partnum}'

mpp-format-int: '{image4k.layout[''root''].partnum}'

dustymabe · 2025-07-31T16:47:16Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

            target: /
          - name: boot
            type: org.osbuild.ext4
            source: disk
            partition:
-              mpp-format-int: '{image4k.layout[''boot''].partnum}'
+              mpp-format-int: '{image.layout[''boot''].partnum}'


Suggested change

mpp-format-int: '{image.layout[''boot''].partnum}'

mpp-format-int: '{image4k.layout[''boot''].partnum}'

dustymabe · 2025-07-31T16:47:27Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

            target: /boot
          - name: efi
            type: org.osbuild.fat
            source: disk
            partition:
-              mpp-format-int: '{image4k.layout[''EFI-SYSTEM''].partnum}'
+              mpp-format-int: '{image.layout[''EFI-SYSTEM''].partnum}'


Suggested change

mpp-format-int: '{image.layout[''EFI-SYSTEM''].partnum}'

mpp-format-int: '{image4k.layout[''EFI-SYSTEM''].partnum}'

dustymabe · 2025-07-31T16:48:06Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

@@ -708,46 +504,14 @@ pipelines:
            type: org.osbuild.xfs
            source: disk
            partition:
-              mpp-format-int: '{image4k.layout[''root''].partnum}'
+              mpp-format-int: '{image.layout[''root''].partnum}'


Suggested change

mpp-format-int: '{image.layout[''root''].partnum}'

mpp-format-int: '{image4k.layout[''root''].partnum}'

dustymabe · 2025-07-31T16:48:14Z

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

            target: /
          - name: boot
            type: org.osbuild.ext4
            source: disk
            partition:
-              mpp-format-int: '{image4k.layout[''boot''].partnum}'
+              mpp-format-int: '{image.layout[''boot''].partnum}'


Suggested change

mpp-format-int: '{image.layout[''boot''].partnum}'

mpp-format-int: '{image4k.layout[''boot''].partnum}'

jlebon · 2025-08-01T21:33:04Z

Some comments.

I think there are a few things we need to iron out before we can really move forward with this:
1. supporting both old and new paths at the same time
Do we need to? Usually when we make a change this large we roll it out slowly, which means we have to support both ways for some time.

This PR is ignoring that fact, but TBH looking at OSBuild configs that support both would be pretty intimidating, so I'm not excited about trying to do that either. I'd be interested in @jlebon or @travier's thoughts.

Rolling this out in FCOS first would be preferable indeed. Instead of adding mpp conditionals, could we just have a separate set of manifests to use temporarily? Not ideal either, I know.

If we're sufficiently convinced by the diff, we could do a hard cut over. The problem is in ensuring the diff captures everything well. I think it helps a lot though that the main changes here are happening at the filesystem level.

openshift-ci bot added the do-not-merge/work-in-progress label Jul 17, 2025

gemini-code-assist bot reviewed Jul 17, 2025

View reviewed changes

dustymabe mentioned this pull request Jul 17, 2025

stages/ignition: parametrize the path to boot osbuild/osbuild#2149

Merged

openshift-merge-robot added the needs-rebase label Jul 18, 2025

dustymabe mentioned this pull request Jul 19, 2025

Use bootc install to-filesystem to build our bootimages coreos/fedora-coreos-tracker#1827

Open

This comment was marked as outdated.

Sign in to view

jbtrystram mentioned this pull request Jul 23, 2025

install to-filesystem: support opt-out for the root and boot mountspec kernel arguments bootc-dev/bootc#1441

Closed

jbtrystram commented Jul 28, 2025

View reviewed changes

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml Outdated Show resolved Hide resolved

jbtrystram force-pushed the osbuild-bootc-install-fs branch from 59f1061 to 254f877 Compare July 29, 2025 09:47

openshift-merge-robot removed the needs-rebase label Jul 29, 2025

jbtrystram force-pushed the osbuild-bootc-install-fs branch 3 times, most recently from cdcb9d0 to bb4270f Compare July 30, 2025 07:35

jbtrystram added 3 commits July 30, 2025 09:38

Add required osbuild patches to use bootc install

310bd60

Theses are patch from the following PRs: osbuild/osbuild#2152 osbuild/osbuild#2149

jbtrystram force-pushed the osbuild-bootc-install-fs branch from bb4270f to 310bd60 Compare July 30, 2025 07:38

jbtrystram marked this pull request as ready for review July 30, 2025 07:44

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 30, 2025

dustymabe reviewed Jul 31, 2025

View reviewed changes

	# To get a shell in the osbuild supervin VM uncomment this.
	# To get a shell in the osbuild supermin VM uncomment this.

	stateroot: fedora-coreos
	stateroot:
	mpp-format-string: '{osname}'

	mpp-format-int: '{image.layout[''root''].partnum}'
	mpp-format-int: '{image4k.layout[''root''].partnum}'

	mpp-format-int: '{image.layout[''boot''].partnum}'
	mpp-format-int: '{image4k.layout[''boot''].partnum}'

	mpp-format-int: '{image.layout[''EFI-SYSTEM''].partnum}'
	mpp-format-int: '{image4k.layout[''EFI-SYSTEM''].partnum}'

osbuild: use bootc install to deploy the container #4224

Are you sure you want to change the base?

osbuild: use bootc install to deploy the container #4224

Uh oh!

Conversation

jbtrystram commented Jul 17, 2025

openshift-ci bot commented Jul 17, 2025

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dustymabe commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

dustymabe commented Jul 19, 2025

dustymabe commented Jul 19, 2025

jbtrystram commented Jul 21, 2025

dustymabe commented Jul 21, 2025

This comment was marked as outdated.

This comment was marked as outdated.

jbtrystram commented Jul 21, 2025

jbtrystram commented Jul 24, 2025

jbtrystram commented Jul 28, 2025

Uh oh!

jbtrystram commented Jul 29, 2025

jbtrystram commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jbtrystram commented Jul 30, 2025

openshift-ci bot commented Jul 30, 2025

dustymabe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlebon commented Aug 1, 2025

dustymabe commented Jul 17, 2025 •

edited

Loading

jbtrystram commented Jul 29, 2025 •

edited

Loading