A production-ready, containerized Elastic Stack deployment with a 3-node Elasticsearch cluster, Kibana, Fleet Server, and Elastic Agent. This stack implementation for auditing logs in distributed environments.

For production deployments on Linux, ensure the host system is properly tuned (e.g., swap disabled, file descriptors increased, and memory locks configured) according to the Elasticsearch System Configuration guide.

To streamline this configuration (check carefully before apply).

• Manual configuration: Use scripts/set_important_es_sysconfig.sh to apply recommended settings.
• Automated configuration: Use the Ansible playbooks for consistent configuration across multiple nodes.

Copy and configure the environment file, then run the setup script:

cp .env.example .env

# Fill some important variables in .env
ELASTIC_PASSWORD=
KIBANA_PASSWORD=

# openssl rand -hex -32
KIBANA_ENCRYPTION_KEY=
KIBANA_REPORTING_KEY=
KIBANA_SECURITY_KEY=
MONGO_DB_USERNAME=
MONGO_DB_PASSWORD=
# MINIO_ROOT_USER=
# MINIO_ROOT_PASSWORD=

bash setup_elastic_stack.sh

The script handles all steps end-to-end: building images, waiting for health checks, creating the Fleet Server service token, fetching the agent enrollment token, and starting Fleet Server and Elastic Agent in the correct order.

Options:

bash setup_elastic_stack.sh --rebuild     # full teardown + rebuild
bash setup_elastic_stack.sh --fleet-only  # redeploy Fleet Server + Agent only

For a step-by-step walkthrough with explanations, see docs/DEPLOYMENT.md.

bash scripts/check_health.sh                # health check all services + verify token type
bash scripts/cleanup_offline_agents.sh      # unenroll all offline agents
bash scripts/rotate_fleet_server_token.sh   # rotate the Fleet Server service token
bash scripts/get_agent_enrollment_token.sh  # fetch and update agent enrollment token
bash scripts/setup_elastic_templates.sh     # apply ILM policies, component templates, and index templates
bash scripts/import_kibana_dashboard.sh     # apply dashboard template

1. Access the Kibana UI at https://localhost:5601.
2. Log in with username elastic and the password configured in your .env file.
3. Navigate to Fleet > Agent policies and select the General Agent Policy.
4. Click Add integration and search for Custom Logs. Configure it (e.g., as a Filestream) to read your application logs. Ensure the dataset or data stream is set to match your expected pattern (e.g., logs-audit-*).

Note on Pre-configurations (CI/CD): If you need to pre-configure Elasticsearch templates or Kibana dashboards, modify the JSON/NDJSON files in the assets/ directory. These are used by the automated setup scripts to bootstrap the environment.

assets/
├── elasticsearch/
│   ├── component-templates/
│   ├── data-stream/
│   ├── ilm-policies/
│   ├── index-templates/
│   └── ingest-pipelines/
└── kibana/
    ├── dashboards/
    ├── index-pattern/
    └── visualizations/

Since the dashboard templates and index configurations are applied automatically during setup, you can immediately begin analyzing data.

Navigate to Analytics > Discover. Select the logs-audit-* data view. You can use KQL (Kibana Query Language) to search and filter incoming log events.

Navigate to Analytics > Dashboards and open the pre-built Audit Logs dashboard. You can customize existing panels or create new visualizations based on your requirements.

To ingest and visualize entirely new types of data beyond the built-in audit logs, from Home to navigate Stack Management and follow the standard Elastic data management workflow:

1. Define Data Rules: First, establish how Elasticsearch should process and store the new data. Create an Index Lifecycle Policy (ILM), Index Template, and (if necessary) an Ingest Pipeline tailored to your new data source.

2. Create a Data View: Make the new indices searchable in Kibana. Navigate to Stack Management > Data Views and create a view matching your new index pattern.

3. Analyze: Return to Analytics > Discover or Dashboards to query the new data and build custom visualizations.

Currently, the S3-compatible snapshot repository setup with MinIO (storage-compose.yml) is a manual post-deployment step. To fully automate this, future development should consider:

• Incorporating the startup of MinIO services (storage-compose.yml) into the main deployment flow.
• Integrating scripts/setup_minio_repository.sh into setup_elastic_stack.sh to automatically inject S3 credentials into the Elasticsearch keystore and register the repository via API.