Inquiry regarding Database Migration failures and Retrieval errors after updates

[sharpdima]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Content

hello Dify Team,
I am managing multiple Dify projects via Docker Compose and I've encountered persistent issues after updating to the latest version. I need your technical insight on the following:

Code Modification vs. Migration: In one project, I manually modified api/core/llm_generator/prompts.py. After a git pull, the worker container began crashing with an ImportError: cannot import name 'LLM_MODIFY_CODE_SYSTEM'. Does manual tampering with internal prompt files directly break the flask db upgrade process?

Dataset Retrieval Errors (The "Text" Project Issue): In another project where I didn't modify the code as much, the containers are running, but the Dataset Test (Retrieval) fails completely. The chatbot responds with "Not found in documents" even though the files are visible in the Knowledge base.

Inconsistent Indexing: It seems that after an update, the existing indexes in Weaviate/PostgreSQL become incompatible with the new API logic. Is it mandatory to re-index (re-upload) all documents after a major version update, or is there a way to repair the existing index via CLI?

Root Cause: Is the "Running migrations" hang-up a result of these code discrepancies? We want to ensure that updating Dify doesn't lead to "data loss" in terms of retrieval functionality.

Best regards

[dosubot[bot]]

Manual changes to internal files like api/core/llm_generator/prompts.py can absolutely break migrations and container startup. If you remove or rename constants such as LLM_MODIFY_CODE_SYSTEM, any code or migration logic that imports them will fail with ImportError, causing the worker to crash and blocking the migration process entirely. The migration CLI (upgrade-db) loads the full app, so any code inconsistency or missing symbol will halt migrations and may leave your database in a partially upgraded state (source).

For retrieval errors like "Not found in documents" after an upgrade, this is often due to vector index incompatibility. Major Dify or Weaviate updates can change the expected schema (for example, requiring a default named vector in Weaviate). Old indexes won't match the new API logic, so retrieval fails even if the files are present in the Knowledge base. The vector configuration in Weaviate is immutable, so you can't patch old collections in place—you must re-index or re-upload documents to rebuild the collections with the correct schema (source).

Dify provides CLI tools for this: use upgrade-db to run database migrations and vdb-migrate to delete and rebuild vector indexes for datasets and apps. These commands do not patch indexes in place—they rebuild them from the current database state. There is no CLI or documented process to repair existing indexes without re-indexing (source).

Migration hang-ups are very often caused by code discrepancies—if your codebase doesn't match the expected schema or logic (for example, due to manual edits), migrations can hang or fail. To avoid data loss and maintain retrieval functionality: avoid manual edits to core files, always test upgrades in a staging environment, back up your data and indexes, and re-index after major version changes (source).

If you still see issues after following these steps, sharing the specific error logs and details of any manual code changes will help pinpoint the problem.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}