DevOps - Vamana-Nestor Pipeline

DevOps automation and orchestration for the Vamana-Nestor intelligent code analysis system.

📊 Pipeline Flow Overview

The Vamana-Nestor system operates in a two-phase pipeline:

graph LR
    subgraph PHASE1[🟢 PHASE 1: EXTRACTION]
        V1[Git Repo +<br/>Confluence]
        V2[Vamana<br/>Multi-Agent<br/>Analysis]
        V3[JSON<br/>Knowledge<br/>Files]
    end
    
    subgraph PHASE2[🔵 PHASE 2: LOAD & QUERY]
        N1[Agent Helper<br/>Load Data]
        N2[Letta Agent<br/>+ Memory]
        N3[Chainlit UI<br/>Chat Interface]
    end
    
    User1[👨‍💻 Developer]
    User2[💡 AI Insights]
    
    V1 --> V2
    V2 --> V3
    V3 --> N1
    N1 --> N2
    N2 --> N3
    
    User1 -->|Initiates| V1
    User1 -->|Queries| N3
    N3 -->|Responds| User2
    
    classDef phase1Style fill:#4CAF50,stroke:#2E7D32,stroke-width:3px,color:#fff
    classDef phase2Style fill:#2196F3,stroke:#1565C0,stroke-width:3px,color:#fff
    classDef userStyle fill:#FF5722,stroke:#D84315,color:#fff
    
    class V1,V2,V3 phase1Style
    class N1,N2,N3 phase2Style
    class User1,User2 userStyle

Phase 1: Extraction (Vamana) 🟢

Input → Process → Store

graph TD
    Input1[📦 Git Repository]
    Input2[📚 Confluence Docs]
    
    Start[🚀 Vamana CrewAI Flow]
    
    subgraph Crews[Parallel Processing]
        LC[Logic Extraction Crew]
        KC[Knowledge Synthesizer Crew]
        
        L1[👤 Logic Extractor Agent]
        L2[👤 Domain Agent]
        
        K1[👤 Doc Analyst Agent]
        K2[👤 Synthesizer Agent]
    end
    
    Output1[domain_details.json]
    Output2[confluence_analysis.json]
    Final[📁 knowledge/output/]
    
    Input1 --> Start
    Input2 --> Start
    Start --> Crews
    
    LC --> L1
    L1 --> L2
    L2 --> Output1
    
    KC --> K1
    K1 --> K2
    K2 --> Output2
    
    Output1 --> Final
    Output2 --> Final
    
    classDef inputStyle fill:#9C27B0,stroke:#6A1B9A,color:#fff
    classDef processStyle fill:#4CAF50,stroke:#2E7D32,color:#fff
    classDef outputStyle fill:#FF9800,stroke:#E65100,color:#fff
    
    class Input1,Input2 inputStyle
    class Start,LC,KC,L1,L2,K1,K2 processStyle
    class Output1,Output2,Final outputStyle

Phase 2: Load & Query (Nestor) 🔵

Load → Configure → Query

graph TD
    Input[📁 knowledge/output/]
    Helper[🧠 Agent Helper<br/>agent_main.py]
    Create[Create Letta Agent]
    
    subgraph Storage[Storage Options]
        DB[(📚 PostgreSQL<br/>Archival Store)]
        Folder[📂 Shared Folder<br/>File-based]
    end
    
    Agent[🤖 Letta Agent + Memory]
    Server[⚙️ Letta Server :8283]
    UI[💬 Chainlit UI :8000]
    Dev[👨‍💻 Developer Queries]
    
    Input --> Helper
    Helper --> Create
    Create --> Storage
    
    DB -.-> Agent
    Folder -.-> Agent
    
    Agent --> Server
    Server --> UI
    UI --> Dev
    
    classDef inputStyle fill:#FF9800,stroke:#E65100,color:#fff
    classDef processStyle fill:#2196F3,stroke:#1565C0,color:#fff
    classDef storageStyle fill:#9C27B0,stroke:#6A1B9A,color:#fff
    classDef userStyle fill:#00BCD4,stroke:#006064,color:#fff
    
    class Input inputStyle
    class Helper,Create,Agent,Server processStyle
    class DB,Folder storageStyle
    class UI,Dev userStyle

🏗️ Architecture Components

Phase 1: Vamana (Extraction Engine)

Component	Technology	Purpose
CrewAI Flow	Python, CrewAI	Orchestrates multi-agent extraction
Logic Extraction Crew	2 Agents	Extracts code logic and domain models
Knowledge Synthesizer Crew	2 Agents	Processes documentation and requirements
Helper Methods	Python	File filtering, git exclusion
Progress GUI	tkinter	Real-time progress visualization

Output: Consolidated JSON files in vamana/src/logic_generator_flow/knowledge/output/

Phase 2: Nestor (Load & Query Engine)

Component	Technology	Purpose
Agent Helper	Python, Letta Client	Manages agent lifecycle
Letta Agent	Letta Framework	Memory-enabled AI agent
Letta Server	REST API (:8283)	Agent orchestration backend
PostgreSQL	Database (:5432)	Persistent knowledge storage
Chainlit UI	Web Interface (:8000)	Conversational interface

Output: Interactive AI assistant with code intelligence

🔄 Complete Pipeline Execution

Step-by-Step Flow

sequenceDiagram
    participant Dev as Developer
    participant Vamana as Vamana Flow
    participant JSON as JSON Storage
    participant Nestor as Nestor Agent
    participant UI as Chainlit UI

    Note over Dev,UI: PHASE 1: EXTRACTION
    Dev->>Vamana: crewai flow kickoff
    Vamana->>Vamana: Scan & analyze codebase
    Vamana->>Vamana: Process documentation
    Vamana->>JSON: Save consolidated JSON

    Note over Dev,UI: PHASE 2: LOAD
    Dev->>Nestor: python agent_main.py
    Nestor->>JSON: Read all JSON files
    Nestor->>Nestor: Create AnalysisAgent
    Nestor->>Nestor: Load to storage

    Note over Dev,UI: PHASE 3: QUERY
    Dev->>UI: chainlit run
    Dev->>UI: Ask questions
    UI->>Nestor: Query knowledge
    Nestor->>UI: AI-powered responses

🚀 Quick Start Commands

Prerequisites

# Vamana environment
conda activate Conversational-Agentic-Pipeline

# Nestor requirements
pip install letta chainlit

Run Phase 1: Extraction

cd /Users/pradip/Development/vamana
crewai flow kickoff
# Output: knowledge/output/*.json files

Run Phase 2: Load & Query

Terminal 1 - Start Letta Server:

letta server
# Server running on http://localhost:8283

Terminal 2 - Create Agent:

cd /Users/pradip/Development/nestor/src/crews/agents
python agent_main.py
# Agent created with knowledge loaded

Terminal 3 - Launch UI:

cd /Users/pradip/Development/nestor/src/crews/chainlit
chainlit run chainlit_conversational_agent.py
# UI available at http://localhost:8000

📁 Project Structure

dev-ops/
├── README.md                      # This file
├── architecture.md                # Complete architecture documentation
├── DIAGRAMS_INDEX.md             # Diagram file mapping
├── architecture-all-*.svg        # Exported architecture diagrams
├── architecture-all-*.png        # PNG versions of diagrams
├── n8n/
│   ├── docker-compose.yml        # n8n automation setup
│   └── workflows/                # Automation workflows
└── scripts/
    ├── run-nestor.sh             # Nestor automation script
    └── run-vamana.sh             # Vamana automation script

📊 Architecture Diagrams

Six comprehensive diagrams have been generated showing the complete system architecture:

High-Level System Architecture - Overall system components and flow
Data Flow Sequence - Step-by-step execution sequence
Vamana Internal Flow - Extraction phase processing
Nestor Agent Creation - Agent setup and configuration
Network Architecture - Ports, protocols, and connections
Component Breakdown - Detailed component organization

View Diagrams: All diagrams are available in both SVG and PNG formats. See DIAGRAMS_INDEX.md for details.

Regenerate Diagrams:

cd /Users/pradip/Development/dev-ops
mmdc -i architecture.md -o architecture-all.svg

🔧 Technology Stack

Phase 1 (Vamana)

Framework: CrewAI (MIT License)
Language: Python 3.11+
Tools: DirectoryReadTool, FileReadTool
Validation: Pydantic (MIT License)
GUI: tkinter (PSF License)
API Integration: atlassian-python-api (Apache 2.0)

Phase 2 (Nestor)

Agent Framework: Letta (Apache 2.0)
UI Framework: Chainlit (Apache 2.0)
Database: PostgreSQL
Client: letta-client

Shared Services

LLM: OpenAI GPT-4o-mini
Embeddings: OpenAI text-embedding-ada-002
Environment: python-dotenv (BSD-3-Clause)

🌐 Network Ports

Service	Port	Protocol	Purpose
Chainlit UI	8000	HTTP/WebSocket	User interface
Letta Server	8283	HTTP/REST	Agent management
PostgreSQL	5432	TCP	Knowledge storage

📈 Performance Metrics

Vamana Processing (50-file repository)

Initialization: ~5 seconds
File Processing: 10-30 seconds per file
Consolidation: 10-15 seconds
Total Time: 20-30 minutes

Nestor Load & Query

Server Start: ~5 seconds
Agent Creation: 2-3 seconds
Data Loading (DB): 60-90 seconds for 150 passages
Data Loading (Folder): 30-45 seconds for 50 files
Query Response: 2-5 seconds per query

🔐 Security Considerations

Authentication

API keys stored in .env files (never commit to git)
Letta token generated per server session
PostgreSQL credentials with limited permissions

Data Protection

Local processing (code never leaves machine except LLM API calls)
HTTPS/TLS for external API communication
File system permissions for access control

📚 Additional Documentation

Vamana README: /Users/pradip/Development/vamana/README.md
Nestor README: /Users/pradip/Development/nestor/src/readme.md
Architecture Details: architecture.md
Diagram Index: DIAGRAMS_INDEX.md

🤝 Contributing

This DevOps pipeline is part of the Vamana-Nestor intelligent code analysis system. For questions or contributions, please refer to the individual project READMEs.

Version: 2.0
Last Updated: November 20, 2025
Status: Production Ready

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
n8n		n8n
ARCHITECTURE.md		ARCHITECTURE.md
README.md		README.md
key-features-slide.md		key-features-slide.md
solution-overview.md		solution-overview.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

DevOps - Vamana-Nestor Pipeline

📊 Pipeline Flow Overview

Phase 1: Extraction (Vamana) 🟢

Phase 2: Load & Query (Nestor) 🔵

🏗️ Architecture Components

Phase 1: Vamana (Extraction Engine)

Phase 2: Nestor (Load & Query Engine)

🔄 Complete Pipeline Execution

Step-by-Step Flow

🚀 Quick Start Commands

Prerequisites

Run Phase 1: Extraction

Run Phase 2: Load & Query

📁 Project Structure

📊 Architecture Diagrams

🔧 Technology Stack

Phase 1 (Vamana)

Phase 2 (Nestor)

Shared Services

🌐 Network Ports

📈 Performance Metrics

Vamana Processing (50-file repository)

Nestor Load & Query

🔐 Security Considerations

Authentication

Data Protection

📚 Additional Documentation

🤝 Contributing

About

Uh oh!

Releases

Packages

Languages

pradipd25/dev-ops

Folders and files

Latest commit

History

Repository files navigation

DevOps - Vamana-Nestor Pipeline

📊 Pipeline Flow Overview

Phase 1: Extraction (Vamana) 🟢

Phase 2: Load & Query (Nestor) 🔵

🏗️ Architecture Components

Phase 1: Vamana (Extraction Engine)

Phase 2: Nestor (Load & Query Engine)

🔄 Complete Pipeline Execution

Step-by-Step Flow

🚀 Quick Start Commands

Prerequisites

Run Phase 1: Extraction

Run Phase 2: Load & Query

📁 Project Structure

📊 Architecture Diagrams

🔧 Technology Stack

Phase 1 (Vamana)

Phase 2 (Nestor)

Shared Services

🌐 Network Ports

📈 Performance Metrics

Vamana Processing (50-file repository)

Nestor Load & Query

🔐 Security Considerations

Authentication

Data Protection

📚 Additional Documentation

🤝 Contributing

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages