feat: add Rerankers from Azure

[Psylockz]

This change adds an Azure Rerank Retriever node to Flowise, allowing reranking via the Azure Foundry endpoint.

Features

• Adds a new AzureRerankRetriever node under Retrievers
• Calls Azure AI Foundry endpoint
• Returns reranked documents with relevance_score in metadata

Implementation details

• New node implementation: packages/components/nodes/retrievers/AzureRerankRetriever/
• Users can implement their own endpoint from Azure Foundry

Usage

Users can now add the Azure Rerank Retriever node to their flows and rerank documents through secure Azure Foundry infrastructure.

Closes #5555

Screenshots:

[gemini-code-assist]

Summary of Changes

Hello @Psylockz, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates Azure AI Foundry's reranking functionality into Flowise by adding a new AzureRerankRetriever node. This allows users to enhance the relevance of retrieved documents by applying advanced semantic reranking, ensuring that the most pertinent information is prioritized and returned with a clear relevance score.

Highlights

• New Azure Rerank Retriever Node: Introduces a dedicated node for document reranking using Azure AI Foundry.
• Azure AI Foundry Integration: Enables seamless connection and utilization of Azure AI Foundry's reranking capabilities.
• Relevance Scoring: Reranked documents now include a 'relevance_score' in their metadata for improved result evaluation.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new Azure Rerank Retriever node. The implementation is largely correct, but I've identified several areas for improvement to enhance robustness, code consistency, and type safety. My feedback for AzureRerank.ts focuses on improving property naming, strengthening type-safety, and replacing silent error handling with explicit error throwing to prevent unexpected behavior. For AzureRerankRetriever.ts, I've suggested adding credential validation to avoid potential runtime errors and correcting an inconsistent variable name to align with the project's coding style.

[Psylockz]

@HenryHengZJ I would really appreciate a review :)

[Psylockz]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a new AzureRerankRetriever node, which is a great addition. The implementation is solid and follows the existing patterns in the codebase. I've provided a few suggestions to improve code conciseness, enhance error handling for better debugging, and ensure consistency in naming conventions. Overall, this is a well-executed feature.

[gemini-code-assist]

The error handling for the API call can be improved to provide more context upon failure. When an axios error occurs, the response body often contains valuable information. Including this in the thrown error message will make debugging significantly easier.

        } catch (error) {
            if (axios.isAxiosError(error) && error.response) {
                throw new Error(`Azure Rerank API call failed with status ${error.response.status}: ${JSON.stringify(error.response.data)}`);
            }
            throw new Error(`Azure Rerank API call failed: ${error.message}`);
        }

[gemini-code-assist]

The logic for processing reranked documents can be simplified. Using map is more concise and idiomatic in TypeScript than using forEach with an external array. Additionally, since the API is called with top_n: this.k, the response should already contain the top k documents, making the call to splice() redundant.

            return returnedDocs.data.results.map((result: { index: number; relevance_score: number }) => {
                const doc = documents[result.index];
                doc.metadata.relevance_score = result.relevance_score;
                return doc;
            });

[gemini-code-assist]

The loop for concatenating document content can be written more concisely using map and join. This improves readability and is a more functional approach.

            const docs = await retriever.getRelevantDocuments(query ? query : input);
            const finaltext = docs.map((doc) => doc.pageContent).join('\n');
            return handleEscapeCharacters(finaltext, false);