Data Pipelines

Overview

Provides the ability to ingest, validate, transform and index (ElasticSearch) arbitrary datasets.

Features

• Give content editors a place to update datasets
• Support for CSV and JSON files
• Support for CSV and JSON URLs
• Developers can declare a pipeline which is a series of transforms and validation rules
• Data is then transformed, validated and output to a destination
• Default output destinations include JSON and CSV files

Configuration/Installation

• Download and install the module like normal
• Follow the readme to define a pipeline in YML
• Visit /admin/content/datasets to add and manage datasets

Requirements

Requires the following modules:

• Entity module

Ecosystem

Destination plugins

https://www.drupal.org/project/data_pipelines_elasticsearch
https://www.drupal.org/project/data_pipelines_opensearch

Source plugins

https://www.drupal.org/project/data_pipelines_sftp

Known problems

Please use the issue queue to report issues.