Make `cosa build` support building "ostree" artifact using podman/buildah #4242

PeaceRebel · 2025-07-29T09:23:44Z

If --with-builda flag is passed or COREOS_ASSEMBLER_BUILD_WITH_BUILDAH=1, build ostree artifacts using buildah. Else, got the traditional way.
Copy source ociarchive in cosa import rather than importing layer by layer.
Force create buildid ref on importing ociarchive

Wants: coreos/fedora-coreos-config#3628 (optional, if de96356 is acceptable)

gemini-code-assist

Code Review

This pull request introduces support for building ostree artifacts using podman/buildah, gated by the COREOS_ASSEMBLER_BUILD_OCI environment variable. The changes also include optimizations for cosa import when dealing with oci-archives and a necessary adjustment to ostree ref creation. The implementation is solid, and I have a suggestion to enhance robustness in the modified code.

src/cmd-import

src/cosalib/cmdlib.py

build.sh

cmd/coreos-assembler.go

jbtrystram

Nice ! Minor comments.
It looks like the second commit message is missing a word?

PeaceRebel · 2025-07-31T15:31:05Z

I'm making another change here to build without the additional volume mounts. With that we can drophttps://github.com/coreos/fedora-coreos-config/pull/3628

dustymabe · 2025-07-31T16:02:17Z

src/cosalib/cmdlib.py

+        # --force is needed for builds from cosa build-oci (using buildah) where
+        # the ref seems to be already present in the repo.
+        subprocess.check_call(['ostree', 'refs', '--repo', tmpd, refs[0], '--create', ref, '--force'])


I'm not sure this is needed. What we need to do is make our versions unique on every build. Right now it's just statically set.

So yeah. If you do consecutive container builds they are going to have conflicts, but we should make our tooling not have that problem.

I was getting conflict on the first build itself.

dustymabe · 2025-07-31T16:06:30Z

Currently cosa build has some more behaviors than what this is reflecting.

You can:

cosa build -> generates ostree/qemu
cosa build ostree -> generates ostree
cosa build ostree qemu openstac -> generates ostree/qemu/openstack (I think)

I'm kind of of the opinion we should change existing cosa build to drop this extra functionality (and deal with the fallout in various CI locations) first to pave way for adding this new method of building, but without having to pollute this new method of building with some of the old cruft.

WDYT?

cmd/coreos-assembler.go

cmd/build-container-native.go

dustymabe · 2025-07-31T16:12:32Z

I'm making another change here to build without the additional volume mounts. With that we can drophttps://github.com/coreos/fedora-coreos-config/pull/3628

Sounds good. I'll review when you're ready

PeaceRebel · 2025-07-31T18:26:09Z

I'm making another change here to build without the additional volume mounts. With that we can drophttps://github.com/coreos/fedora-coreos-config/pull/3628

I have pushed it a separate commit here 9f09123
If this is good, I'll squash it.

PeaceRebel · 2025-07-31T18:33:47Z

Currently cosa build has some more behaviors than what this is reflecting.

You can:
1. `cosa build` -> generates ostree/qemu

2. `cosa build ostree` -> generates ostree

3. `cosa build ostree qemu openstac` -> generates ostree/qemu/openstack (I think)
I'm kind of of the opinion we should change existing cosa build to drop this extra functionality (and deal with the fallout in various CI locations) first to pave way for adding this new method of building, but without having to pollute this new method of building with some of the old cruft.

WDYT?

We can provide the same functionality with this new way. We can check if there are additional arguments except ostree & container and pass it on to the "old" script to generate other artifacts. Is that something we want?

PeaceRebel · 2025-07-31T19:30:30Z

/gemini review

dustymabe · 2025-07-31T19:31:14Z

cmd/build-with-buildah.go

-		"-v", "/srv/tmp:/run/tmp",
-		"-t", fmt.Sprintf("oci-archive:%s/out.ociarchive", tempdir),
-		"src/config"}
+	podmanCmd := []string{"/usr/lib/coreos-assembler/build-with-buildah.sh", fmt.Sprintf("%s/out.ociarchive", tempdir)}


/usr/lib/coreos-assembler/build-with-buildah.sh

I don't see this script added to this commit, but I will say at the same time that I did like it the way it was before with the entire command in this file :)

Ah. Missed it when I renamed it.

but I will say at the same time that I did like it the way it was before with the entire command in this file :)

The problem with those changes is that we might need to modify Containerfile to as you see in the related PR. This makes it the podman command from the Containerfiles's comments or the GH Actions to fails.

Is there any other way to run multiple command in the supermin vm?

Removed the script and added all command in the go file.

gemini-code-assist

Code Review

This pull request introduces a new build path for cosa build using buildah, triggered by a new --with-buildah flag or an environment variable. The changes include a new Go function to orchestrate the build, logic to switch to this new path, and an optimization for importing local OCI archives.

My review focuses on improving the Go code to follow language idioms and fixing a logic bug in how the new build path is triggered. The bug could cause command-line arguments to be silently ignored, leading to unexpected build behavior. The suggestions for the Go code aim to improve maintainability by using standard library functions for path manipulation.

gemini-code-assist · 2025-07-31T19:32:59Z

cmd/coreos-assembler.go

+	mustEnvOption, ok := os.LookupEnv("COREOS_ASSEMBLER_BUILD_WITH_BUILDAH")
+	// We don't expect any other arguments
+	// Possible to make better by checking if '--native' in argv.
+	return (len(argv) == 1 && argv[0] == "--with-buildah") || (ok && mustEnvOption == "1")


The current logic for determining whether to use the buildah path has a flaw. If the COREOS_ASSEMBLER_BUILD_WITH_BUILDAH environment variable is set to 1, mustBuildWithBuildah will return true even if other command-line arguments (like --force) are provided. These arguments are then silently ignored because buildWithBuildah() doesn't process them. This could lead to unexpected behavior.

The buildah path should only be taken if --with-buildah is the sole argument, or if the environment variable is set and no arguments are provided. This aligns with the help text that states 'Cannot be used with other options'.

Suggested change

return (len(argv) == 1 && argv[0] == "--with-buildah") || (ok && mustEnvOption == "1")

return (len(argv) == 1 && argv[0] == "--with-buildah") || (ok && mustEnvOption == "1" && len(argv) == 0)

I think this is ok. Ignoring.

cmd/build-with-buildah.go

dustymabe · 2025-07-31T19:34:10Z

We can provide the same functionality with this new way. We can check if there are additional arguments except ostree & container and pass it on to the "old" script to generate other artifacts. Is that something we want?

ehh. Like I said earlier, I actually think the existing behavior isn't ideal from a code organiziation perspective, but it is nice from a dev UX perspective.

If we did try to keep at least the current default behavior I certainly wouldn't call out to the existing cmd-build, but rather just run cmd-osbuild qemu here.

Again. I think we should bite the bullet and drop the current behavior IMO.

jcapiitao · 2025-08-01T06:11:23Z

/retest

If build subcommand is called with `--native` flag, build ostree artifacts using buildah. Else, got the traditional way.

Import command can do a simple of ociarchive instead of importing each layer. Output of cosa build using podman/buildah is an ociarchive and this can be leveraged there.

Builds from cosa using podman/buildah seems to existing ref of the same name we create.

Changes in Containerfile fedora-coreos-config can be drop with this.

dustymabe · 2025-08-01T16:47:35Z

src/cmd-build

+  --with-buildah        Use the container runtime (buildah) to build ostree and container artifacts.
+                        Cannot be used with other options.


so what's weird here is we are adding to the documentation to the cmd-build but this is an option src/cmd-build doesn't actually support.

--with-buildah is supported by cmd/build-container-native.go and switched on at the cmd/coreos-assembler.go level.

It's OK as a PoC to support it only in cmd/build-container-native.go but also what about all of these other options?

For example, --versionary is really useful in the context of making our versions unique on each build and --autolock=VERSION is required for both bodhi testing and downstream for RHCOS multi-arch builds.

I think more likely what we'd need to do is do away with the switch that happens in cmd/coreos-assembler.go and then just shell out here to cmd-supermin-run based on --with-buildah.

Of course, this could happen in a followup PR, but would probably be required before say we could switch any of our pipelines over to actually using this code.

jlebon · 2025-08-01T20:36:45Z

OK yeah, this is somewhat my fault. cmd-build is full of tricky bash code and I was hoping to avoid having to touch it at all by just making a separate e.g. cosa buildoci and intercepting early on in coreos-assembler.go whether to redirect cosa build to that instead based on the env var.

But... yes we need at least the versionary bits and the autolock stuff. The latter would be obsoleted longer term once we move to hermetic builds in Konflux.

The versionary bits happen quite early in cmd-build. We'd have to move up the autolock generation. And then cmd-build could just exec cosa buildoci [--version $VERSION] [--lockfile $AUTOLOCK]?

jlebon · 2025-08-01T20:45:20Z

Another big thing cmd-build does which is still relevant to the container native flow is change detection. But the way that should be implemented I think will look different (e.g. using hermetic builds + git, or having truly reproducible builds). There's also a lot of crud built up over the years in there (e.g. references of kickstart).

I think a goal once we've cut over is that cosa build is just a very minimal wrapper around podman build. Things that truly must happen in a "setup phase" (e.g. versionary) can be done there, but otherwise we should push all the complexity into the container build itself.

gemini-code-assist bot reviewed Jul 29, 2025

View reviewed changes

src/cmd-import Outdated Show resolved Hide resolved

PeaceRebel mentioned this pull request Jul 29, 2025

Containerfile: Make use of cosa directories coreos/fedora-coreos-config#3628

Draft

travier reviewed Jul 29, 2025

View reviewed changes

src/cosalib/cmdlib.py Outdated Show resolved Hide resolved

jbtrystram reviewed Jul 29, 2025

View reviewed changes

build.sh Outdated Show resolved Hide resolved

jbtrystram reviewed Jul 29, 2025

View reviewed changes

cmd/coreos-assembler.go Outdated Show resolved Hide resolved

jbtrystram reviewed Jul 29, 2025

View reviewed changes

PeaceRebel force-pushed the pr/build-oci branch from 5ea144c to 7a3abe3 Compare July 29, 2025 12:29

dustymabe reviewed Jul 31, 2025

View reviewed changes

cmd/coreos-assembler.go Outdated Show resolved Hide resolved

dustymabe reviewed Jul 31, 2025

View reviewed changes

cmd/build-container-native.go Outdated Show resolved Hide resolved

PeaceRebel force-pushed the pr/build-oci branch 3 times, most recently from 2fccb66 to c1fb721 Compare July 31, 2025 19:20

PeaceRebel requested review from dustymabe and jlebon July 31, 2025 19:22

dustymabe reviewed Jul 31, 2025

View reviewed changes

gemini-code-assist bot reviewed Jul 31, 2025

View reviewed changes

PeaceRebel force-pushed the pr/build-oci branch from c1fb721 to cba1e5e Compare July 31, 2025 19:34

PeaceRebel force-pushed the pr/build-oci branch from cba1e5e to d490bc1 Compare July 31, 2025 20:19

PeaceRebel added 3 commits August 1, 2025 13:49

cosa build ostree artifacts using podman

a2a9187

If build subcommand is called with `--native` flag, build ostree artifacts using buildah. Else, got the traditional way.

cmd/import: copy file if image source is ociarchive

e2ddf68

Import command can do a simple of ociarchive instead of importing each layer. Output of cosa build using podman/buildah is an ociarchive and this can be leveraged there.

cmd/import: Force create an ostree ref when importing ociarchive

b3e9130

Builds from cosa using podman/buildah seems to existing ref of the same name we create.

PeaceRebel force-pushed the pr/build-oci branch from d490bc1 to 0dc9389 Compare August 1, 2025 08:19

PeaceRebel added 2 commits August 1, 2025 17:50

Build without needing to mount additional volumes

508bfc3

Changes in Containerfile fedora-coreos-config can be drop with this.

cmd/build: Add --with-buildah to help text

9223f47

PeaceRebel force-pushed the pr/build-oci branch from 0dc9389 to 9223f47 Compare August 1, 2025 12:20

PeaceRebel requested a review from dustymabe August 1, 2025 12:21

dustymabe reviewed Aug 1, 2025

View reviewed changes

jlebon mentioned this pull request Aug 1, 2025

jobs/build: support importing OCI image using cosa import and skipping build stages coreos/fedora-coreos-pipeline#1185

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make `cosa build` support building "ostree" artifact using podman/buildah #4242

Make `cosa build` support building "ostree" artifact using podman/buildah #4242

PeaceRebel commented Jul 29, 2025 •

edited

Loading

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbtrystram left a comment

PeaceRebel commented Jul 31, 2025

dustymabe Jul 31, 2025

PeaceRebel Jul 31, 2025

dustymabe Jul 31, 2025

dustymabe commented Jul 31, 2025

Uh oh!

Uh oh!

dustymabe commented Jul 31, 2025

PeaceRebel commented Jul 31, 2025

PeaceRebel commented Jul 31, 2025 •

edited

Loading

PeaceRebel commented Jul 31, 2025

dustymabe Jul 31, 2025

PeaceRebel Jul 31, 2025

PeaceRebel Jul 31, 2025 •

edited

Loading

PeaceRebel Aug 1, 2025

gemini-code-assist bot left a comment

gemini-code-assist bot Jul 31, 2025

PeaceRebel Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

dustymabe commented Jul 31, 2025

jcapiitao commented Aug 1, 2025

dustymabe Aug 1, 2025

jlebon commented Aug 1, 2025

jlebon commented Aug 1, 2025

	return (len(argv) == 1 && argv[0] == "--with-buildah") \|\| (ok && mustEnvOption == "1")
	return (len(argv) == 1 && argv[0] == "--with-buildah") \|\| (ok && mustEnvOption == "1" && len(argv) == 0)

		--with-buildah Use the container runtime (buildah) to build ostree and container artifacts.
		Cannot be used with other options.

Make cosa build support building "ostree" artifact using podman/buildah #4242

Are you sure you want to change the base?

Make cosa build support building "ostree" artifact using podman/buildah #4242

Conversation

PeaceRebel commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbtrystram left a comment

Choose a reason for hiding this comment

PeaceRebel commented Jul 31, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustymabe commented Jul 31, 2025

Uh oh!

Uh oh!

dustymabe commented Jul 31, 2025

PeaceRebel commented Jul 31, 2025

PeaceRebel commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PeaceRebel commented Jul 31, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PeaceRebel Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

gemini-code-assist bot Jul 31, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dustymabe commented Jul 31, 2025

jcapiitao commented Aug 1, 2025

Choose a reason for hiding this comment

jlebon commented Aug 1, 2025

jlebon commented Aug 1, 2025

Make `cosa build` support building "ostree" artifact using podman/buildah #4242

Make `cosa build` support building "ostree" artifact using podman/buildah #4242

PeaceRebel commented Jul 29, 2025 •

edited

Loading

PeaceRebel commented Jul 31, 2025 •

edited

Loading

PeaceRebel Jul 31, 2025 •

edited

Loading