Problem/Motivation

One of Drupal's underpinnings and something that is often seen as unchangeable is the way in which Drupal enables extensions (themes/modules). This can happen at runtime and is expected to take immediate effect. With "at runtime" in this case I mean that any in-progress work (e.g. a request, cron job, or Drupal CLI command) should automatically adapt to these changes. This capability brings with it a lot of complexity around some of the key primitives that underpin Drupal: the service container, the module-/theme-registry, the database schema, available file system streams, and the router.

This issue is meant as a thought experiment and as a way to discuss this invariant. In Drupal's past we've already seen a shift in how modules that are enabled get managed. With the introduction of things such as configuration management we see that module installation already happens outside of the production runtime of the application.

It is not my intention to eliminate the ability for a site manager to enable a module through the UI. That is likely too useful in people's journey of discovering Drupal, or for local development.

However, I would like to propose a world where enabling a module in such a manner no longer has immediate effect. What if the current request would complete as if that module is not yet enabled and any side effects would only take effect on the next request. I would argue that 80% of the things that are used in a request (the last 20% being entity data and perhaps some to be identified issues) can function perfectly fine after being initially loaded into memory.

This model is not without precedent. Blue/green deployment practices demonstrate that deferring the effect of changes to the next unit of work, rather than applying them mid-flight, is a viable and widely used approach. The technical details of that model are included as an appendix below for those interested.

Proposed resolution

I would like to use this issue to explore what benefits this "next-request" model brings and what new problems it causes that we would need to solve to get there. Specifically, I'm hoping the community can help weigh in on the following:

  • Is the premise sound? Is mid-request module enabling a problem worth solving, or are the current trade-offs acceptable?
  • Does the 80/20 claim hold up? What cases exist where the in-memory state of a request genuinely cannot function correctly without immediately reflecting a module being enabled?
  • Are there other things in Drupal that this might make a lot easier or has further performance benefits?
  • What breakage cases can others identify that this model would introduce?
  • What would it take to provide developers with the right primitives to declare mutations to shared state and ensure they execute at the correct point?

Identified benefits

  • Drupal core code can be simplified by no longer having to reload fundamental services mid-task
  • Many expensive to calculate items that depend on which extensions are enabled could be pre-calculated and cached on disk/in OpCache (e.g. the service container, all plugin caches)
  • Organizations running Drupal would, through blue/green-deployments, gain new capabilities for high uptime guarantees

Identified challenges to be solved

  • Provide developers with the primitives to declare mutations to shared state and ensure that they can be executed on the right side of the deployment.
  • To be successful some things that may currently depend on config (e.g. whether a route or plugin is registered) may need a runtime check of that config, if the item is used, rather than rebuilding the cache when the config changes.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Blue/green deployments

Blue/Green deployments are very simple in concept: any change being made to a database or shared resource should be compatible with the old and new version of the application. This is already similar in Drupal to how a module must be uninstalled before the files on disk that make up the module can be removed.

That is, in blue/green deployment we should be careful to differentiate between "local state" and "shared state". Where local state are the things that are loaded into a process's memory that affect its behavior. Shared state are things that should persist between requests (such as user data).

Local state includes the instructions (source code) of the applications and it may be different. As long as applications with different local states can both interact with the "shared state" without breaking.

Many of Drupal's core primitives, such as the service container, the module/theme registry, available file system streams, and the router, can already be considered as "local state". If we update their stored representation (either on disk or in the database) but do not change the in-memory representation, then behaviorally not much would change for the current process even if other processes now see a different set of modules being enabled.

Changes to the API of a shared state (e.g. the database schema) can be classified into two categories: additive (e.g. adding a column or table) and subtractive (removing a column or table). The strategy of dealing with these is simple in theory.

Additive changes should be made while the previous version of the application is running. Once the changes are made the new version of the application that relies on these changes can start running.

Subtractive changes should be made after the previous version of the application has stopped. This means that the new version of the application should be started first and no longer use the shared state that will be removed.

Thus,
Additive:
- update state
- deploy

Subtractive:
- deploy
- update state

AI disclaimer: The text in this issue is my own. I've had help from Claude to provide the structure which moved the diversion of Blue/Green deployments to the appendix, so that this is first-and-foremost about feasibility/benefits for Drupal, rather than specific techniques.

Comments

kingdutch created an issue. See original summary.

catch’s picture

catch
he/him
Primary language English
commented

The biggest limitation here would be that we're no longer able to enable multiple modules at a time, e.g. in the installer, during configuration syncs, and in kernel tests. Possibly this could be ameliorated if there's a manual way to trigger the refresh that is currently done automatically, but then the complexity might not get saved then. For manually installing modules via the UI, we could probably do this with a 1 by 1 batch (or use AJAX/HTMX instead of a massive form with checkboxes) so it's not really a big deal there.

We already save container refreshes until a module specifies it needs one, so maybe it could still be multiple if a new request happens when a container refresh is required but it all starts to get tricky.

There are also other operations that change the container, like switching from monolingual to multilingual and one or two other config changes, those could probably be adapted so they affect things only from the next request too though.

nicxvan’s picture

nicxvan commented
Component: other » extension system
nicxvan’s picture

nicxvan commented

That would still be runtime too, should the title say during a single request?

catch’s picture

catch
he/him
Primary language English
commented
Title: [meta] Re-evaluate whether module/theme install/uninstall should happen during runtime » [meta] Re-evaluate whether module/theme install/uninstall should take effect during a single request

Tried a re-title.

kingdutch’s picture

kingdutch
he/him
Primary language English
commented
Title: [meta] Re-evaluate whether module/theme install/uninstall should take effect during a single request » [meta] Re-evaluate whether module/theme install/uninstall should happen during runtime

I'm going to change the title back because I explicitly didn't want to limit the discussion to requests.

With the work being done around the Drupal CLI or with async, I hope that having long running processes that might be performing multiple tasks (either as continuous background tasks or perhaps for websocket connections) become a reality.

The way I see such a thing working would be that there's something in Drupal which can update the shared state representations of the things affected by a module (e.g. updating the container_dump.php file). That could be a specific web request or a command line invocation such as dr pm:install. Importantly it should not affect processes that are ongoing or other web requests that are already in flight.

In that sense I also don't entirely understand why multiple modules might be problematic. I suppose that perhaps in the category of problems to be solved, would be that modules currently expect their dependencies to be enabled when they get installed and the code that runs when a module gets installed is a free for all. If the solution to that problem would be a stricter description of the things that can happen during a module install (e.g. "update schema", "update config", "generate content" – each of which can already be described by the recipe system at this point; or figured out by the config importer), then the process responsible for performing that enabling could ensure the proper timing of those operations.

Perhaps the better title to describe what I want to achieve if runtime is confusing things would be something along the lines of: "Freeze container and extension registry representation at process/request start".

The outside representation for a new process/request would be allowed to change, but the in-process version of it would not. That would enable the optimizations of in-process immutability (such as loading the representation from OPCache or having a tailored autoloader that can be loaded from disk and not needing to do any run-time class resolution).