• Features
  • Document Chunking
  • Local LLM Model Support
  • LaTeX Support
  • Multi-lingual Support
  • Extended File Format Support
• Configuration
• Constraints & Limitations
• Request Parameters
• Installation
• Environment Setup
• Usage
  • Supported File Types
  • Chunking Strategies
  • Credential Options
• Development Setup
• Contributing
• License
• Community and Support
• Connect with Us

Supported File Types: PDF, DOCX, PPTX, HTML, Markdown, Images, Webpages

Chunking Strategies:

• Fixed Size : Splits text into chunks of specified size with optional overlap
• Page-based : Splits PDF by pages (PDF only, falls back to paragraph for other file types)
• Semantic : Uses YOLO for segmentation and VLM + OCR for intelligent extraction of text, images, and tables — followed by semantic grouping for clean, contextual output
• Paragraph : Splits text by paragraphs
• Heading : Splits text by identified headings
• Hierarchical : Advanced multi-level chunking with parent-child relationships

• Modular LLM Selection: Choose from multiple LLM providers and models
• Local Model Integration: Support for locally hosted models (Ollama)
• Provider Options: OpenAI, Anthropic, Google, Cohere, and custom endpoints
• Model Flexibility: Switch between different models for different chunking strategies

• Mathematical Equations: Full LaTeX rendering and processing
• Scientific Documents: Optimized for academic and technical papers
• Formula Extraction: Intelligent extraction and preservation of mathematical formulas
• Equation Chunking: Maintains mathematical context across chunks

• Language Detection: Automatic language identification
• Parameterized Processing: Language-specific chunking strategies
• Unicode Support: Full support for non-Latin scripts
• Localized Chunking: Language-aware paragraph and sentence boundaries

• Images : JPG, PNG, TIFF, BMP with OCR capabilities
• Webpages : Direct URL processing with content extraction
• Spreadsheets : Excel, CSV with structured data extraction

• OPENAI_API_KEY: Your OpenAI API key for semantic chunking

💡 Tip: We recommend installing in a virtual environment for project isolation.

# Create a virtual environment (optional but recommended)
python -m venv venv

# Activate the virtual environment
# On macOS/Linux:
source venv/bin/activate
# On Windows:
# venv\Scripts\activate

# Install unsiloed-parser
pip install unsiloed-parser

unsiloed-parser requires Python 3.8 or higher and has the following dependencies:

Core Dependencies:

• openai - OpenAI API integration
• PyPDF2 - PDF processing
• python-docx - Word document processing
• python-pptx - PowerPoint processing
• Pillow - Image processing
• pytesseract - OCR capabilities
• aiohttp - Async HTTP client
• requests - HTTP library
• beautifulsoup4 - HTML parsing
• validators - URL validation

AI & ML:

• ultralytics - YOLO model integration
• opencv-python-headless - Computer vision
• numpy - Numerical computing

Utilities:

• python-dotenv - Environment variable management
• markdown - Markdown processing
• lxml - XML/HTML parsing
• html2text - HTML to text conversion
• pdf2image - PDF to image conversion

Before using unsiloed-parser, set up your OpenAI API key for semantic chunking:

# Linux/macOS
export OPENAI_API_KEY="your-api-key-here"

# Windows (Command Prompt)
set OPENAI_API_KEY=your-api-key-here

# Windows (PowerShell)
$env:OPENAI_API_KEY="your-api-key-here"

Create a .env file in your project directory:

OPENAI_API_KEY=your-api-key-here

Then in your Python code:

from dotenv import load_dotenv
load_dotenv()  # This loads the variables from .env

import os
import Unsiloed

result = Unsiloed.process_sync({
    "filePath": "./test.pdf",
    "credentials": {
        "apiKey": os.environ.get("OPENAI_API_KEY")
    },
    "strategy": "semantic",
    "chunkSize": 1000,
    "overlap": 100
})

print(result)

Example Output (Semantic Chunking):

{
  "file_type": "pdf",
  "strategy": "semantic",
  "total_chunks": 98,
  "avg_chunk_size": 305.05,
  "chunks": [
    {
      "text": "Introduction to Machine Learning",
      "metadata": {
        "page_number": 1,
        "semantic_group_index": 0,
        "element_count": 1,
        "primary_element_type": "Section-header",
        "avg_confidence": 0.92,
        "combined_bbox": [100, 50, 500, 120],
        "strategy": "semantic_openai_boundary_detection",
        "split_confidence": 0.95,
        "reading_order_start": 0,
        "reading_order_end": 0,
        "constituent_elements": [
          {
            "element_type": "Section-header",
            "bbox": [100, 50, 500, 120],
            "reading_order": "page_1_element_0"
          }
        ]
      }
    },
    {
      "text": "Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience...",
      "metadata": {
        "page_number": 1,
        "semantic_group_index": 1,
        "element_count": 2,
        "primary_element_type": "Text",
        "avg_confidence": 0.89,
        "combined_bbox": [100, 150, 800, 400]
      }
    },
    {
      "text": "The image shows a neural network architecture with multiple layers...",
      "metadata": {
        "page_number": 2,
        "semantic_group_index": 2,
        "primary_element_type": "Picture",
        "avg_confidence": 0.94,
        "combined_bbox": [100, 200, 700, 600]
      }
    },
    {
      "text": "```markdown\n| Model | Accuracy | Speed |\n|-------|----------|-------|\n| CNN   | 95%      | Fast  |\n```",
      "metadata": {
        "page_number": 3,
        "semantic_group_index": 3,
        "primary_element_type": "Table",
        "primary_content_type": "table",
        "avg_confidence": 0.91,
        "combined_bbox": [150, 100, 850, 500]
      }
    }
  ]
}

import Unsiloed

html_result = Unsiloed.process_sync({
    "filePath": "./webpage.html",
    "credentials": {
        "apiKey": os.environ.get("OPENAI_API_KEY")
    },
    "strategy": "paragraph"
})

import Unsiloed

markdown_result = Unsiloed.process_sync({
    "filePath": "./README.md",
    "credentials": {
        "apiKey": os.environ.get("OPENAI_API_KEY")
    },
    "strategy": "heading"
})

import Unsiloed

url_result = Unsiloed.process_sync({
    "filePath": "https://example.com",
    "credentials": {
        "apiKey": os.environ.get("OPENAI_API_KEY")
    },
    "strategy": "paragraph"
})

import asyncio
import Unsiloed

async def async_processing():
    result = await Unsiloed.process({
        "filePath": "./test.pdf",
        "credentials": {
            "apiKey": os.environ.get("OPENAI_API_KEY")
        },
        "strategy": "semantic"
    })
    return result

# Run async processing
async_result = asyncio.run(async_processing())

import Unsiloed
import os

try:
    result = Unsiloed.process_sync({
        "filePath": "./document.pdf",
        "credentials": {
            "apiKey": os.environ.get("OPENAI_API_KEY")
        },
        "strategy": "semantic"
    })
    print(f"Successfully processed {len(result['chunks'])} chunks")
    
except FileNotFoundError:
    print("Error: File not found. Please check the file path.")
    
except ValueError as e:
    print(f"Error: Invalid configuration - {e}")
    
except Exception as e:
    print(f"An unexpected error occurred: {e}")

Transform any document format into LLM-ready chunks with intelligent parsing and extraction.

SEO Keywords: PDF chunking for RAG, OCR for images, document parsing for LLM, semantic document chunking, AI-powered text extraction, webpage to text conversion, DOCX parsing, structured data extraction

Choose the optimal strategy for your document processing needs and RAG pipeline.

💡 Performance Tips:

• Use semantic for best RAG results and AI-powered content understanding
• Use fixed for consistent embedding sizes and token management
• Use heading for technical documentation and structured content
• Use hierarchical for complex documents requiring context preservation

You can provide credentials in three ways:

1. In the options (recommended):

result = Unsiloed.process_sync({
    "filePath": "./test.pdf",
    "credentials": {
        "apiKey": "your-openai-api-key"
    },
    "strategy": "semantic"
})

2. Environment variable:

export OPENAI_API_KEY="your-openai-api-key"

3. Using .env file:

OPENAI_API_KEY=your-openai-api-key

• Python 3.8 or higher
• pip (Python package installer)
• git

1. Clone the repository:

git clone https://github.com/Unsiloed-AI/Unsiloed-Parser.git
cd Unsiloed-Parser

2. Create a virtual environment:

# Using venv
python -m venv venv

# Activate the virtual environment
# On Windows
venv\Scripts\activate
# On macOS/Linux
source venv/bin/activate

3. Install dependencies:

pip install -r requirements.txt

4. Set up your environment variables:

# Create a .env file
echo "OPENAI_API_KEY=your-api-key-here" > .env

5. Run the FastAPI server locally (if applicable):

uvicorn Unsiloed.app:app --reload

6. Access the API documentation: Open your browser and go to http://localhost:8000/docs

We welcome contributions to unsiloed-parser!

Here's how you can help:

1. Fork the repository and clone your fork:

git clone https://github.com/YOUR_USERNAME/Unsiloed-Parser.git
cd Unsiloed-Parser

2. Install development dependencies:

pip install -r requirements.txt

1. Create a new branch for your feature:

git checkout -b feature/your-feature-name

2. Make your changes and write tests if applicable

3. Commit your changes:

git commit -m "Add your meaningful commit message here"

4. Push to your fork:

git push origin feature/your-feature-name

5. Create a Pull Request from your fork to the main repository

• We follow PEP 8 for Python code style
• Use type hints where appropriate
• Document functions and classes with docstrings
• Write tests for new features

This project is licensed under the Apache-2.0 License - see the LICENSE file for details.

• GitHub Discussions 💬: For questions, ideas, and discussions
• Issues 🐛: For bug reports and feature requests
• Pull Requests 🔧: For contributing to the codebase

• Star ⭐ the repository to show support
• Watch 👀 for notification on new releases

📧 Email Us Get in touch directly hello@unsiloed-ai.com

📅 Schedule a Call Book a discovery session Schedule with our team

🌐 Visit Our Website Explore more features www.unsiloed-ai.com