Dind for k8s and others

[MartinSchmidt]

Hi, I was having an issue with running Dind with the k8s backend when having to use testcontainers, since it really needs localhost because of the many dynamic ports that are needed.
There is a current issue #3977 where the discussion about sidecars have begun, and I created a draft PR #5666 that has added support for it.

I am looking for a discussion on the topic, and I am willing to put in some time on the work.

BUT, I was thinking if this should be something more native, so that we could create a "feature" (or something else) one could enable in project settings, that allows priviledged dind, but not priviledged in generel.

This feature should then be configured from env variables on the server, and could work different for the different backends.

On the k8s backend, it would then run it as a sidecar, mount the default volume, and map env variables to point to tls certs and everything, really lowering the barrier of entry, and I really believe docker within CI should be a first class citizen since so many different tooling these days do tests, builds and the like in this manner.

I have also been looking into if podman was a better solution, but I know both would probably be needed, but there is a nice guide (albiet a few years old) on how to run podman in k8s in different privileges.

Suggested way

steps:
  - name: test
    image: docker:cli
    features:
      - type: dind
    commands:
      - docker version

current way

steps:
  - name: test
    image: docker:cli # use 'docker:<major-version>-cli' or similar in production
    environment:
      DOCKER_HOST: 'tcp://docker:2376'
      DOCKER_CERT_PATH: '/dind-certs/client'
      DOCKER_TLS_VERIFY: '1'
    volumes:
      - /opt/woodpeckerci/dind-certs:/dind-certs
    commands:
      - docker version

services:
  - name: docker
    image: docker:dind # use 'docker:<major-version>-dind' or similar in production
    privileged: true
    environment:
      DOCKER_TLS_CERTDIR: /dind-certs
    volumes:
      - /opt/woodpeckerci/dind-certs:/dind-certs
    ports:
      - 2376

[qwerty287]

I can't really tell you something about the general things of your point, so I just comment on this paragraph:

BUT, I was thinking if this should be something more native, so that we could create a "feature" (or something else) one could enable in project settings, that allows priviledged dind, but not priviledged in generel.

Personally I don't really like to have "problem-specific" solutions. So I'd rather keep this general (similar to how it's now).

But I could also imagine that this is a problem many users have, so maybe there are many people suggesting this as well. I'm open for the discussion of course.

About the actual implementation I think that for the docker backend you just have to give it the privileged flag. Right now, if you set the "Security" trusted flag, this is actually the only feature that is allowed. So actually, for the docker backend, your idea is possible already (it's generic though so this could change if we add other features behind the security flag).

[MartinSchmidt]

I agree that solutions should be as general as possible, as it keeps the code more clean and lean. That is also why I looked at the solution for adding sidecars to the k8s backend, as a general solution.

This problem might also mostly be one for k8s, since the problem ain't as complicated for the other backends, and I might have found a more general solution.

I have looked into plugins, specifically PLUGINS_PRIVILEGED so I can see I can have some of this feature there, so each repo don't require priviledged them selves, buns can be configured for a docker-in-docker plugin.

I have also looked at the example extension, where I could implement so a keyword can be replaced with the sidecar running the dind plugin, so the only missing part is really the sidecar for k8s in the PR #5666 any chance to get some comments on that before finishing it?