Skip to content

Conversation

@utsav-bx
Copy link

@utsav-bx utsav-bx commented Sep 15, 2025

Problem

Fixes #5126 - Flowise hangs indefinitely when Redis connections drop, causing ECONNRESET errors during queue processing.

Solution

This PR implements a two-part fix to prevent indefinite hangs:

1. Redis Connection Resilience (Optional Configuration)

Adds configurable Redis connection settings via environment variables without hardcoded defaults:

  • REDIS_CONNECT_TIMEOUT - Connection timeout in milliseconds
  • REDIS_COMMAND_TIMEOUT - Command timeout in milliseconds
  • REDIS_MAX_RETRIES_PER_REQUEST - Maximum number of retries per request
  • REDIS_LAZY_CONNECT - Enable lazy connection (true/false)

These settings are only applied if explicitly set, allowing Redis to use its own defaults otherwise.

2. Job Execution Timeouts (Core Fix)

Adds JOB_TIMEOUT environment variable (default: 300000ms / 5 minutes) and implements explicit timeouts on all job.waitUntilFinished() calls across:

  • buildChatflow.ts
  • agentflowv2-generator/index.ts
  • documentstore/index.ts (5 occurrences)
  • nodes/index.ts
  • upsertVector.ts

This prevents jobs from hanging indefinitely when Redis connections fail.

Key Changes

  • Enhanced packages/server/src/queue/QueueManager.ts with conditional Redis resilience settings
  • Updated packages/server/.env.example with new configuration options
  • Added explicit timeouts to all 9 waitUntilFinished() calls
  • Maintains full backward compatibility with existing Redis environment variables

Addresses PR Feedback

  • ✅ Removed hardcoded defaults per @HenryHengZJ's feedback
  • ✅ Uses conditional property spreading - only includes settings when env vars are defined
  • ✅ Allows Redis to use its own internal defaults when not configured

Testing

  • ✅ Build passes with pnpm build
  • ✅ TypeScript compilation successful
  • ✅ All existing Redis env vars preserved
  • ✅ Backward compatible - works with or without new env vars
utsav-bx pushed a commit to utsav-bx/Flowise that referenced this pull request Nov 15, 2025
Addresses PR FlowiseAI#5219 feedback to add Redis connection resilience settings:
- connectTimeout: Configurable via REDIS_CONNECT_TIMEOUT
- commandTimeout: Configurable via REDIS_COMMAND_TIMEOUT
- maxRetriesPerRequest: Configurable via REDIS_MAX_RETRIES_PER_REQUEST
- lazyConnect: Configurable via REDIS_LAZY_CONNECT

Settings are only included when explicitly set via environment variables,
allowing Redis to use its own internal defaults when not configured.
Also refactored keepAlive to use the same conditional spreading pattern.
utsav-bx pushed a commit to utsav-bx/Flowise that referenced this pull request Nov 15, 2025
Addresses PR FlowiseAI#5219 feedback to add Redis connection resilience settings:
- connectTimeout: Configurable via REDIS_CONNECT_TIMEOUT
- commandTimeout: Configurable via REDIS_COMMAND_TIMEOUT
- maxRetriesPerRequest: Configurable via REDIS_MAX_RETRIES_PER_REQUEST
- lazyConnect: Configurable via REDIS_LAZY_CONNECT

Settings are only included when explicitly set via environment variables,
allowing Redis to use its own internal defaults when not configured.
Also refactored keepAlive to use the same conditional spreading pattern.
utsav-bx and others added 4 commits November 15, 2025 23:32
- Add Redis timeout and retry configuration via environment variables
- Fix waitUntilFinished() hanging indefinitely on Redis connection drops
- Add connectTimeout, commandTimeout, maxRetriesPerRequest settings
- Maintain backward compatibility with existing Redis env vars
- Resolves issue FlowiseAI#5126

Breaking change: None
Addresses PR FlowiseAI#5219 feedback to add Redis connection resilience settings:
- connectTimeout: Configurable via REDIS_CONNECT_TIMEOUT
- commandTimeout: Configurable via REDIS_COMMAND_TIMEOUT
- maxRetriesPerRequest: Configurable via REDIS_MAX_RETRIES_PER_REQUEST
- lazyConnect: Configurable via REDIS_LAZY_CONNECT

Settings are only included when explicitly set via environment variables,
allowing Redis to use its own internal defaults when not configured.
Also refactored keepAlive to use the same conditional spreading pattern.
Completes fix for issue FlowiseAI#5126 by adding explicit timeout parameters
to all waitUntilFinished() calls across the codebase.

Changes:
- Added JOB_TIMEOUT environment variable (default: 300000ms / 5 minutes)
- Applied timeout to waitUntilFinished in:
  - buildChatflow.ts (prediction queue)
  - agentflowv2-generator/index.ts (prediction queue)
  - upsertVector.ts (upsert queue)
  - nodes/index.ts (prediction queue - custom functions)
  - documentstore/index.ts (5 locations - upsert queue)
- Updated .env.example with documentation for JOB_TIMEOUT and Redis resilience settings

This prevents indefinite blocking when Redis fails or jobs hang,
addressing the root cause where requests would return 504 timeouts
with no automatic recovery.
@utsav-bx utsav-bx force-pushed the bugfix/redis-connection-timeout-fix branch from a33cf59 to 761a13a Compare November 15, 2025 23:33
@utsav-bx utsav-bx requested a review from HenryHengZJ November 17, 2025 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants