Skip to content

Conversation

@jameslamb
Copy link
Member

I've been using dask-cloudprovider a lot recently to test things and come up with reproducible examples.

Through this experience, I learned that ECS / Fargate only allow task definitions to choose from a finite list of CPU and memory settings. This is an AWS limitation and not a dask-cloudprovider one, but it took me a bit of investigation to figure that out.

botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: No Fargate configuration exists for given values.

This PR proposes adding a link to the valid config values for ECS in the docs, so that when people search the dask-cloudprovider docs or Google for this error, they quickly find the answer and the set of valid values.

Thanks for your time and consideration.

Copy link
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great thanks @jameslamb.

Should we also mention this (or link to the troubleshooting guide) from the docstrings that describe these kwargs.

scheduler_cpu: int (optional)
The amount of CPU to request for the scheduler in milli-cpu (1/1024).
Defaults to ``1024`` (one vCPU).
scheduler_mem: int (optional)
The amount of memory to request for the scheduler in MB.
Defaults to ``4096`` (4GB).

worker_cpu: int (optional)
The amount of CPU to request for worker tasks in milli-cpu (1/1024).
Defaults to ``4096`` (four vCPUs).
worker_mem: int (optional)
The amount of memory to request for worker tasks in MB.
Defaults to ``16384`` (16GB).

@jameslamb
Copy link
Member Author

oh sure, good idea! I'll link to the troubleshooting guide from there, to avoid duplicating that AWS link into multiple places.

@jameslamb
Copy link
Member Author

Alright I just pushed an update, let me know what you think! I regenerated the docs like this:

pip install .[digitalocean]
cd doc
make html

Locally, after that change the link looks like this:

image

And I clicked through and confirmed that the relevant reference like /troubleshooting.html#invalid-cpu-or-memory works as expected.

@jacobtomlinson
Copy link
Member

Perfect thanks @jameslamb

@jacobtomlinson jacobtomlinson merged commit fad780e into dask:master Nov 9, 2020
@jacobtomlinson jacobtomlinson added the documentation Improvements or additions to documentation label Nov 9, 2020
@jameslamb jameslamb deleted the docs/ecs-config branch November 9, 2020 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

2 participants