[META] Finalize the Migration system

Under API issues, I separated out some hot topics revolving around migrations as configuration entities. These are blockers for at least the work I'm doing on migrate_upgrade and migrate_plus. They're also mutually interdependent, so we need a battle plan to address them.

#2462233: Migrations should not use the configuration entity system - @alexpott, please chastise me if I've misunderstood, but based on IRC discussion I understand Alex's current position to be that it is OK for migration to be configuration entities as long as we address the pollution problems from migrate_drupal installing all the things into active configuration. #2463909: Migrations should support non-installed default configurations (templates) is my proposal to deal with that. The current model used by migrate_drupal is that all possible core migrations from Drupal 6 and Drupal 7 are automatically installed into active configuration - these migrations are not actually usable as initially installed, they need source database configuration. The runners will select those corresponding to the specific version being upgraded from, inject the source database information, and run those migrations.

Stepping back a bit, I see three broad categories of migration use cases:

1. A completely hard-coded migration (typical for a custom single-site migration). All migration configuration for such a migration can live in a custom module's config/install, or deployed from dev->staging->production environments, and be ready to go.
2. A framework for a class of migrations, meant to be extended for particular site migrations. Current examples of this use case are D7's migrate_d2d and wordpress_migrate, where classes provide default field mappings and the like, which are instantiated as necessary and extended with site-specific mappings per selections made through a UI or overrides in a derived class. In D8 it would be convenient for default configurations to be provided in YAML files, but not installed into active configuration until fully configured in the context of a particular migration project.
3. Completely dynamic migration - ultimately, it would be great to have a general UI where you just point at a database/XML feed/directory of CSVs and build migrations entirely from scratch.

Right now migrate_drupal thinks it's use case 1, but it's really use case 2. Even in the straight upgrade scenario, it needs that source database information, and it also should be filtering the migrations it runs down to those that make sense in the particular site's migration (e.g., book migration should only be attempted if the book module is enabled on both the source and destination sides). And we also would like to support more flexible Drupal-to-Drupal migration scenarios in contrib, leveraging the core migrate_drupal as much as possible - this is solidly use case 2.

The model I see migrate_drupal using is described in #2463909: Migrations should support non-installed default configurations (templates), I won't repeat it here. I will point out that it depends on having #2302307: Support shared configuration between migrate groups as well.

I suggest the order of tackling these issues should be:

1. #2302307: Support shared configuration between migrate groups - this doesn't really depend on the others.
2. #2463909: Migrations should support non-installed default configurations (templates) - the biggest change.
3. #2460529: Migrations need to use the configuration entity dependency system - the other issues, which include some group-based dependency handling, and the big goal of getting the default configs out of config/install, address much of what's in the current patch. There's still some necessary dependency calculation going on here, though - once the other two issues are committed, we can see exactly what should be kept in this one.

Will the Migrate Drupal module handle (a) Drupal 6 CCK node reference fields, and (b) Views 2 setups?

1. The Drupal 7 References module could upgrade Drupal 6 nodereference and userreference fields, but there did not seem to be a way to convert to the preferred Entity reference field type, available in Drupal 7, and required in Drupal 8.
2. Views 6.x-2.x is used by 6 times more people than Views 6.x-3.x. But there is a bug preventing an upgrade between the two.

Without both of these, people will find it hard to migrate to Drupal 8.

Good news on node reference. I would urge work on D6 Views 2, as it has 6 times as many users as D6 Views 3. The schemas don't look too dissimilar.

Is there somewhere better to put in a feature request for Drupal 6 Views 2 migration? A separate issue?

@iantresman - yes, Views 2 support should be a distinct issue.

As suggested, I've posted a request to provide a migration from Views 6.x-2.x to Drupal 8.

alexpott and I had a discussion about this at DrupalCon Los Angeles, some notes:

Looking at the big picture for the Migrate in Core roadmap, there are 6 main areas:

1. Migration groups
2. Config entities / template system
3. Move configurations out of migrate_drupal and into their associated core modules (for example, move block related migrations to core/modules/block - this will also help with dependencies(?)
4. Method for maintaining source DBs for automated tests.
5. D7->D8 migrations
6. Ongoing D6->D8 bug squashing.

In order to move forward, we need to discuss, decide, and act on items 2, 3, and 4 (the first item, migration groups, is close to completion) before we can realistically move on to D7->D8 stuff as they will have wide-ranging effects (especially items 2 and 3). The order of items I have listed above is not necessarily the order we should tackle things:

• items 1 and 4 are mostly independent
• items 5 and 6 should definitely come after 1, 2, 3, and 4
• it may make more sense to tackle item 3 before item 2

alexpott is going to join our weekly call on May 27, I suppose it would be good if we could get a lot of discussion done on the relevant issues so that call ends up being a confirmation call rather than a trip to the bikeshed.

Links to the various issues are listed in the issue summary - Mike Ryan's comment 4 has some good information as well.

-mike

Metadata massaging.

Adding IDs to various headings so they can be linked to, moving done stuff to "Complete," removing unrealistic target dates, etc.

Radically shortened this list to just what's left to remove the remaining Migrate criticals. All the credits to @mikeryan.

@benjy: Yeah, this is pretty far out of date. +1 to closing it.