Skip to content

humn-ai/tf-mod-aws-emr-cluster

 
 

Repository files navigation

README Header

tf-mod-aws-emr-cluster

Module description

Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS.

Usage

IMPORTANT: The master branch is used in source just as an example. In your code, do not pin to master because there may be breaking changes between releases. Instead pin to the release tag (e.g. ?ref=tags/x.y.z) of one of our latest releases.

provider "aws" {
  region = "us-east-2"
}

module "vpc" {
  source  = "cloudposse/vpc/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"

  ipv4_primary_cidr_block = "172.19.0.0/16"

  context = module.label.context
}

module "subnets" {
  source = "cloudposse/dynamic-subnets/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"

  availability_zones   = var.availability_zones
  vpc_id               = module.vpc.vpc_id
  igw_id               = [module.vpc.igw_id]
  ipv4_cidr_block      = [module.vpc.vpc_cidr_block]
  nat_gateway_enabled  = false
  nat_instance_enabled = false

  context = module.label.context
}

module "s3_log_storage" {
  source = "cloudposse/s3-log-storage/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  
  region        = var.region
  namespace     = var.namespace
  stage         = var.stage
  name          = var.name
  attributes    = ["logs"]
  force_destroy = true
}

module "aws_key_pair" {
  source = "cloudposse/key-pair/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  namespace           = var.namespace
  stage               = var.stage
  name                = var.name
  attributes          = ["ssh", "key"]
  ssh_public_key_path = var.ssh_public_key_path
  generate_ssh_key    = var.generate_ssh_key
}

module "emr_cluster" {
  source = "cloudposse/emr-cluster/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  
  namespace                                      = var.namespace
  stage                                          = var.stage
  name                                           = var.name
  master_allowed_security_groups                 = [module.vpc.vpc_default_security_group_id]
  slave_allowed_security_groups                  = [module.vpc.vpc_default_security_group_id]
  region                                         = var.region
  vpc_id                                         = module.vpc.vpc_id
  subnet_id                                      = module.subnets.private_subnet_ids[0]
  route_table_id                                 = module.subnets.private_route_table_ids[0]
  subnet_type                                    = "private"
  ebs_root_volume_size                           = var.ebs_root_volume_size
  visible_to_all_users                           = var.visible_to_all_users
  release_label                                  = var.release_label
  applications                                   = var.applications
  configurations_json                            = var.configurations_json
  core_instance_group_instance_type              = var.core_instance_group_instance_type
  core_instance_group_instance_count             = var.core_instance_group_instance_count
  core_instance_group_ebs_size                   = var.core_instance_group_ebs_size
  core_instance_group_ebs_type                   = var.core_instance_group_ebs_type
  core_instance_group_ebs_volumes_per_instance   = var.core_instance_group_ebs_volumes_per_instance
  master_instance_group_instance_type            = var.master_instance_group_instance_type
  master_instance_group_instance_count           = var.master_instance_group_instance_count
  master_instance_group_ebs_size                 = var.master_instance_group_ebs_size
  master_instance_group_ebs_type                 = var.master_instance_group_ebs_type
  master_instance_group_ebs_volumes_per_instance = var.master_instance_group_ebs_volumes_per_instance
  create_task_instance_group                     = var.create_task_instance_group
  log_uri                                        = format("s3n://%s/", module.s3_log_storage.bucket_id)
  key_name                                       = module.aws_key_pair.key_name
}

Makefile Targets

Available targets:

  help                                Help screen
  help/all                            Display help for all targets
  help/short                          This help short screen
  lint                                Lint terraform code

Requirements

Name Version
terraform >= 0.12.0
terraform >= 0.14.0
aws ~> 2.0
aws >= 3.5.0
null ~> 2.0
template ~> 2.0

Providers

Name Version
aws ~> 2.0 >= 3.5.0

Modules

Name Source Version
label git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_alb git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_core git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_ec2 git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_ec2_autoscaling git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_emr git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_master git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_master_managed git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_service_managed git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_slave git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_slave_managed git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0
label_task git::https://github.com/cloudposse/terraform-null-label.git tags/0.25.0

Resources

Name Type
aws_emr_cluster.default resource
aws_emr_instance_group.task resource
aws_iam_instance_profile.ec2 resource
aws_iam_role.ec2 resource
aws_iam_role.ec2_autoscaling resource
aws_iam_role.emr resource
aws_iam_role_policy_attachment.ec2 resource
aws_iam_role_policy_attachment.ec2_autoscaling resource
aws_iam_role_policy_attachment.emr resource
aws_lb.default resource
aws_lb_listener.http_redirect resource
aws_lb_listener.https resource
aws_lb_target_group.default resource
aws_lb_target_group_attachment.default resource
aws_route53_record.default resource
aws_security_group.alb resource
aws_security_group.managed_master resource
aws_security_group.managed_service_access resource
aws_security_group.managed_slave resource
aws_security_group.master resource
aws_security_group.slave resource
aws_security_group_rule.alb_egress resource
aws_security_group_rule.alb_http_ingress resource
aws_security_group_rule.alb_https_ingress resource
aws_security_group_rule.managed_master_egress resource
aws_security_group_rule.managed_master_service_access_ingress resource
aws_security_group_rule.managed_service_access_egress resource
aws_security_group_rule.managed_slave_egress resource
aws_security_group_rule.master_egress resource
aws_security_group_rule.master_ingress_cidr_blocks resource
aws_security_group_rule.master_ingress_security_groups resource
aws_security_group_rule.slave_egress resource
aws_security_group_rule.slave_ingress_cidr_blocks resource
aws_security_group_rule.slave_ingress_security_groups resource
aws_vpc_endpoint.vpc_endpoint_s3 resource
aws_iam_policy_document.assume_role_ec2 data source
aws_iam_policy_document.assume_role_emr data source
aws_instances.emr_master_instances data source
aws_partition.current data source

Inputs

Name Description Type Default Required
additional_info A JSON string for selecting additional features such as adding proxy information. Note: Currently there is no API to retrieve the value of this argument after EMR cluster creation from provider, therefore Terraform cannot detect drift from the actual EMR cluster if its value is changed outside Terraform string null no
additional_master_security_group The id of the existing additional security group that will be used for EMR master node. If empty, a new security group will be created string "" no
additional_slave_security_group The id of the existing additional security group that will be used for EMR core & task nodes. If empty, a new security group will be created string "" no
additional_tag_map Additional tags for appending to tags_as_list_of_maps. Not added to tags. map(string) {} no
alb_allow_http_access If true, allow ALB traffic to port 80. bool false no
alb_allow_https_access If true, allow ALB traffic to port 443. bool false no
alb_enabled If true, create an Elastic Load Balancer. bool false no
alb_internal If true, create an internal ALB. bool false no
alb_target_group_port The port for the ALB to route traffic to. string "80" no
alb_target_group_protocol The protocol for the ALB to use. string "HTTP" no
alb_target_group_target_type The type of target for the ALB to route traffic to. string "instance" no
applications A list of applications for the cluster. Valid values are: Flink, Ganglia, Hadoop, HBase, HCatalog, Hive, Hue, JupyterHub, Livy, Mahout, MXNet, Oozie, Phoenix, Pig, Presto, Spark, Sqoop, TensorFlow, Tez, Zeppelin, and ZooKeeper (as of EMR 5.25.0). Case insensitive list(string) n/a yes
attributes Additional attributes (e.g. 1) list(string) [] no
auto_termination_idle_timeout Auto termination policy idle timeout in seconds (60 - 604800 supported) string null no
bootstrap_action List of bootstrap actions that will be run before Hadoop is started on the cluster nodes
list(object({
path = string
name = string
args = list(string)
}))
[] no
certificate_arn The ARN of the ACM certificate for SSL encryption. string "" no
configurations_json A JSON string for supplying list of configurations for the EMR cluster. See https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html for more details string "" no
context Single object for setting entire context at once.
See description of individual variables for details.
Leave string and numeric variables as null to use default value.
Individual variable settings (non-null) override settings in context object,
except for attributes, tags, and additional_tag_map, which are merged.
object({
enabled = bool
namespace = string
environment = string
stage = string
name = string
delimiter = string
attributes = list(string)
tags = map(string)
additional_tag_map = map(string)
regex_replace_chars = string
label_order = list(string)
id_length_limit = number
})
{
"additional_tag_map": {},
"attributes": [],
"delimiter": null,
"enabled": true,
"environment": null,
"id_length_limit": null,
"label_order": [],
"name": null,
"namespace": null,
"regex_replace_chars": null,
"stage": null,
"tags": {}
}
no
core_instance_group_autoscaling_policy String containing the EMR Auto Scaling Policy JSON for the Core instance group string null no
core_instance_group_bid_price Bid price for each EC2 instance in the Core instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances string null no
core_instance_group_ebs_iops The number of I/O operations per second (IOPS) that the Core volume supports number null no
core_instance_group_ebs_size Core instances volume size, in gibibytes (GiB) number n/a yes
core_instance_group_ebs_type Core instances volume type. Valid options are gp2, io1, standard and st1 string "gp2" no
core_instance_group_ebs_volumes_per_instance The number of EBS volumes with this configuration to attach to each EC2 instance in the Core instance group number 1 no
core_instance_group_instance_count Target number of instances for the Core instance group. Must be at least 1 number 1 no
core_instance_group_instance_type EC2 instance type for all instances in the Core instance group string n/a yes
create_task_instance_group Whether to create an instance group for Task nodes. For more info: https://www.terraform.io/docs/providers/aws/r/emr_instance_group.html, https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html bool false no
create_vpc_endpoint_s3 Set to false to prevent the module from creating VPC S3 Endpoint bool true no
custom_ami_id A custom Amazon Linux AMI for the cluster (instead of an EMR-owned AMI). Available in Amazon EMR version 5.7.0 and later string null no
delimiter Delimiter to be used between namespace, environment, stage, name and attributes.
Defaults to - (hyphen). Set to "" to use no delimiter at all.
string null no
ebs_root_volume_size Size in GiB of the EBS root device volume of the Linux AMI that is used for each EC2 instance. Available in Amazon EMR version 4.x and later number 10 no
ec2_autoscaling_role_enabled If set to false, will use existing_ec2_autoscaling_role_arn for an existing EC2 autoscaling IAM role that was created outside of this module bool true no
ec2_autoscaling_role_permissions_boundary The Permissions Boundary ARN to apply to the EC2 Autoscaling Role. string "" no
ec2_role_enabled If set to false, will use existing_ec2_instance_profile_arn for an existing EC2 IAM role that was created outside of this module bool true no
ec2_role_permissions_boundary The Permissions Boundary ARN to apply to the EC2 Role. string "" no
emr_role_permissions_boundary The Permissions Boundary ARN to apply to the EMR Role. string "" no
enabled Set to false to prevent the module from creating any resources bool null no
environment Environment, e.g. 'uw2', 'us-west-2', OR 'prod', 'staging', 'dev', 'UAT' string null no
existing_ec2_autoscaling_role_arn ARN of an existing EC2 autoscaling role to attach to the cluster string "" no
existing_ec2_instance_profile_arn ARN of an existing EC2 instance profile string "" no
existing_service_role_arn ARN of an existing EMR service role to attach to the cluster string "" no
id_length_limit Limit id to this many characters.
Set to 0 for unlimited length.
Set to null for default, which is 0.
Does not affect id_full.
number null no
keep_job_flow_alive_when_no_steps Switch on/off run cluster with no steps or when all steps are complete bool true no
kerberos_ad_domain_join_password The Active Directory password for ad_domain_join_user. Terraform cannot perform drift detection of this configuration. string null no
kerberos_ad_domain_join_user Required only when establishing a cross-realm trust with an Active Directory domain. A user with sufficient privileges to join resources to the domain. Terraform cannot perform drift detection of this configuration. string null no
kerberos_cross_realm_trust_principal_password Required only when establishing a cross-realm trust with a KDC in a different realm. The cross-realm principal password, which must be identical across realms. Terraform cannot perform drift detection of this configuration. string null no
kerberos_enabled Set to true if EMR cluster will use kerberos_attributes bool false no
kerberos_kdc_admin_password The password used within the cluster for the kadmin service on the cluster-dedicated KDC, which maintains Kerberos principals, password policies, and keytabs for the cluster. Terraform cannot perform drift detection of this configuration. string null no
kerberos_realm The name of the Kerberos realm to which all nodes in a cluster belong. For example, EC2.INTERNAL string "EC2.INTERNAL" no
key_name Amazon EC2 key pair that can be used to ssh to the master node as the user called hadoop string null no
label_order The naming order of the id output and Name tag.
Defaults to ["namespace", "environment", "stage", "name", "attributes"].
You can omit any of the 5 elements, but at least one must be present.
list(string) null no
log_uri The path to the Amazon S3 location where logs for this cluster are stored string null no
managed_master_security_group The id of the existing managed security group that will be used for EMR master node. If empty, a new security group will be created string "" no
managed_slave_security_group The id of the existing managed security group that will be used for EMR core & task nodes. If empty, a new security group will be created string "" no
master_allowed_cidr_blocks List of CIDR blocks to be allowed to access the master instances list(string) [] no
master_allowed_security_groups List of security group ids to be allowed to connect to the master instances list(string) [] no
master_dns_name (Required) - Name of the ALB record to create in the parent DNS zone specified by zone_id. string n/a yes
master_instance_group_bid_price Bid price for each EC2 instance in the Master instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances string null no
master_instance_group_ebs_iops The number of I/O operations per second (IOPS) that the Master volume supports number null no
master_instance_group_ebs_size Master instances volume size, in gibibytes (GiB) number n/a yes
master_instance_group_ebs_type Master instances volume type. Valid options are gp2, io1, standard and st1 string "gp2" no
master_instance_group_ebs_volumes_per_instance The number of EBS volumes with this configuration to attach to each EC2 instance in the Master instance group number 1 no
master_instance_group_instance_count Target number of instances for the Master instance group. Must be at least 1 number 1 no
master_instance_group_instance_type EC2 instance type for all instances in the Master instance group string n/a yes
name Solution name, e.g. 'app' or 'jenkins' string null no
namespace Namespace, which could be your organization name or abbreviation, e.g. 'eg' or 'cp' string null no
regex_replace_chars Regex to replace chars with empty string in namespace, environment, stage and name.
If not set, "/[^a-zA-Z0-9-]/" is used to remove all characters other than hyphens, letters and digits.
string null no
region AWS region string n/a yes
release_label The release label for the Amazon EMR release. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html string "emr-5.25.0" no
route_table_id Route table ID for the VPC S3 Endpoint when launching the EMR cluster in a private subnet. Required when subnet_type is private string "" no
scale_down_behavior The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized string null no
security_configuration The security configuration name to attach to the EMR cluster. Only valid for EMR clusters with release_label 4.8.0 or greater. See https://www.terraform.io/docs/providers/aws/r/emr_security_configuration.html for more info string null no
service_access_security_group The id of the existing additional security group that will be used for EMR core & task nodes. If empty, a new security group will be created string "" no
service_role_enabled If set to false, will use existing_service_role_arn for an existing IAM role that was created outside of this module bool true no
slave_allowed_cidr_blocks List of CIDR blocks to be allowed to access the slave instances list(string) [] no
slave_allowed_security_groups List of security group ids to be allowed to connect to the slave instances list(string) [] no
stage Stage, e.g. 'prod', 'staging', 'dev', OR 'source', 'build', 'test', 'deploy', 'release' string null no
step_concurrency_level The number of steps that can be executed concurrently. You can specify a maximum of 256 steps. Only valid for EMR clusters with release_label 5.28.0 or greater. number null no
steps List of steps to run when creating the cluster.
list(object({
name = string
action_on_failure = string
hadoop_jar_step = object({
args = list(string)
jar = string
main_class = string
properties = map(string)
})
}))
[] no
subnet_id VPC subnet ID where you want the job flow to launch. Cannot specify the cc1.4xlarge instance type for nodes of a job flow launched in a Amazon VPC string n/a yes
subnet_type Type of VPC subnet ID where you want the job flow to launch. Supported values are private or public string "private" no
tags Additional tags (e.g. map('BusinessUnit','XYZ') map(string) {} no
task_instance_group_autoscaling_policy String containing the EMR Auto Scaling Policy JSON for the Task instance group string null no
task_instance_group_bid_price Bid price for each EC2 instance in the Task instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances string null no
task_instance_group_ebs_iops The number of I/O operations per second (IOPS) that the Task volume supports number null no
task_instance_group_ebs_optimized Indicates whether an Amazon EBS volume in the Task instance group is EBS-optimized. Changing this forces a new resource to be created bool false no
task_instance_group_ebs_size Task instances volume size, in gibibytes (GiB) number 10 no
task_instance_group_ebs_type Task instances volume type. Valid options are gp2, io1, standard and st1 string "gp2" no
task_instance_group_ebs_volumes_per_instance The number of EBS volumes with this configuration to attach to each EC2 instance in the Task instance group number 1 no
task_instance_group_instance_count Target number of instances for the Task instance group. Must be at least 1 number 1 no
task_instance_group_instance_type EC2 instance type for all instances in the Task instance group string null no
termination_protection Switch on/off termination protection (default is false, except when using multiple master nodes). Before attempting to destroy the resource when termination protection is enabled, this configuration must be applied with its value set to false bool false no
use_existing_additional_master_security_group If set to true, will use variable additional_master_security_group using an existing security group that was created outside of this module bool false no
use_existing_additional_slave_security_group If set to true, will use variable additional_slave_security_group using an existing security group that was created outside of this module bool false no
use_existing_managed_master_security_group If set to true, will use variable managed_master_security_group using an existing security group that was created outside of this module bool false no
use_existing_managed_slave_security_group If set to true, will use variable managed_slave_security_group using an existing security group that was created outside of this module bool false no
use_existing_service_access_security_group If set to true, will use variable service_access_security_group using an existing security group that was created outside of this module bool false no
visible_to_all_users Whether the job flow is visible to all IAM users of the AWS account associated with the job flow bool true no
vpc_id VPC ID to create the cluster in (e.g. vpc-a22222ee) string n/a yes
zone_id Route53 parent zone ID. If provided (not empty), the module will create sub-domain DNS records for the masters and slaves string null no

Outputs

Name Description
cluster_id EMR cluster ID
cluster_name EMR cluster name
ec2_role Role name of EMR EC2 instances so users can attach more policies
master_host Name of the cluster CNAME record for the master nodes in the parent DNS zone
master_public_dns Master public DNS
master_security_group_id Master security group ID
slave_security_group_id Slave security group ID

Help

Got a question? We got answers.

File a Github issue, or message us on Slack

Contributors

Erik Osterman
Erik Osterman
Andriy Knysh
Andriy Knysh
Igor Rodionov
Igor Rodionov
Callum Robertson
Callum Robertson
Lawrence Warren
Lawrence Warren


README Footer

About

Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • HCL 97.4%
  • Makefile 2.6%