fix(marshmallow): add compatibility layer for Flask-AppBuilder with marshmallow 4.x

[kevin-lann]

SUMMARY

Upgrades marshmallow version to >= 4.0

• Flask-AppBuilder 5.0.0 auto-generates schema fields for SQLAlchemy relationships that don't exist in marshmallow 4.x's stricter field validation, causing KeyError during schema initialization.
• Our solution patches marshmallow's Schema._init_fields method to detect missing fields that Flask-AppBuilder 5.0.0 auto-generates, and then adds these missing fields dynamically as raw fields with appropriate flags & retry initialization until all missing fields are resolved.
• So, for example, it can handle multiple fields in a sequence (permission_id, view_menu_id, db_id, chart_id, dashboard_id, user_id). All of this logic for this bullet point is encapsulated in the marshmallow_fix.py module, whose job is to specifically address the Flask-AppBuilder 5.0.0 incompatibility with marshmallow 4.x, and then superset_config.py applies the patch by importing it from marshmallow_fix.py.
• All changes in files other than superset_config.py and marshmallow_fix.py are minor convention/syntax changes that are required for the Marshmallow upgrade to >= 4.0.0 (See https://marshmallow.readthedocs.io/en/stable/upgrading.html for more info)

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

BEFORE: Upon pinning marshmallow>=4 in pyproject.tomland initializing the superset app, an initialization error occurs.

AFTER: App runs without error

TESTING INSTRUCTIONS

Start superset application, see that no marshmallow related errors show up.

Run tests with docker compose run --rm superset python -m pytest tests/unit_tests/test_marshmallow_compatibility.py -v

Unit tests created:

1. test_patch_marshmallow_for_flask_appbuilder_applies_patch - Verifies the patch function correctly modifies the Schema class
2. test_patch_functionality_with_real_schema_creation - Tests actual schema creation works with the patch
3. test_patch_handles_schema_with_no_fields - Ensures empty schemas work properly
4. test_raw_field_creation_and_configuration - Validates Raw field configuration
5. test_print_function_can_be_mocked - Tests logging functionality can be tested
6. test_keyerror_exception_handling - Verifies KeyError exception handling
7. test_schema_declared_fields_manipulation - Tests field manipulation capabilities
8. test_flask_appbuilder_field_names_list - Validates the field names our fix handles
9. test_patch_function_is_callable - Ensures the patch function works correctly
10. test_marshmallow_schema_basic_functionality - Confirms basic marshmallow functionality still works

ADDITIONAL INFORMATION

• Has associated issue: Fixes issue Make Superset compatible with marshmallow>=4.0.0 #33162
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

Credit to

@Austin-X @Eyang0612 @PDOracle @MasahisaSekita

[korbit-ai]

Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment /korbit-review.

Your admin can change your review schedule in the Korbit Console