[Feature] Metadata addition to embeddings created using Knowledge Base Source Classes

[sabaul]

Add file metadata information while embedding files using Knowledge Base Source Classes

• Add per-chunk metadata in all KnowledgeSources (filepath, chunk_index, source_type)
• Backwards compatible storage with _coerce_to_records

Reason why this was required

• If we decide to persist the embeddings by creating a child of KnowledgeStorage.py (using the one time embedding way in the knowledge base best practices), it was very difficult to get any file level information.
• This change will add some file level metadata to the embeddings, which can be later used to do chromadb level query and embedding deletion as well (if required).
• This will enable file deletion possibility with the corresponding embedding deletion as well.

Related to #3812

[cursor]

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

[cursor]

Bug: Metadata Corruption in Document Conversion

The zip call in add() uses strict=False, which can silently misalign filepath metadata. If document conversion yields fewer documents than input files, chunks may be incorrectly associated with file paths, corrupting the intended file-level metadata.

 

[cursor]

Bug: Empty String Handling Inconsistency

The _coerce_to_records function skips dictionary-based documents with empty string content due to the if not content: continue check. This creates an inconsistency with string-based inputs, which accept empty content, and can lead to silent data loss, breaking backward compatibility.

 

[cursor]

Bug: Incomplete Flag Causes Dynamic Import Error

The DOCLING_AVAILABLE flag is incomplete, as it doesn't account for DocumentConverter. This allows an ImportError to occur during model_post_init when dynamically importing DocumentConverter, even if DOCLING_AVAILABLE is True.

Additional Locations (1)

• lib/crewai/src/crewai/knowledge/source/crew_docling_source.py#L69-L71