Run on AWS

Create a runner on AWS

To run your models on AWS, you first need to get hold of an EC2 virtual machine (VM). If you already have one, check the requirements below to make sure your VM has sufficient resources.

The second step is to attach your VM runner to Dotscience.

Get an AWS Ubuntu VM.

  1. Go to the AWS management console. If you have an AWS account, sign in, or else create an account and then sign in.

  2. When you are logged into AWS, click on Services on the top navigation. Under Compute, select EC2.

  3. Click Launch instance to see a list of available EC2 instances.

  4. Now choose an EC2 instance to launch.

(a) Select a machine image. In order to run Dotscience, we recommend choosing a recent release of Ubuntu Server as your operating system.

(b) Given your selected machine image, select an instance type. This decision will be based on your memory and CPU requirements, but must meet the following criteria:

  • The instance type should be at least t2.small (1 vCPU, 2GB of RAM) to meet the memory requirements for running Docker.
  • We recommend that you increase the capacity of the disc from the default 8GB to 30GB, so that there is enough storage for the Docker images you’ll use. To do this, click Configure instance details, go to the tab marked Add storage and edit the value in the Size column. Note that you do not want to add a volume here: just increase the root disc size.

VM requirements: 64-bit Ubuntu Server, t2.small or larger instance, 30GB disk at least.

(d) Click Review and Launch to see a summary, then click Launch on that page. Follow the prompt to either create a key pair, or use an existing one. The key pair allows you to connect to your instance securely. If creating a new one, make sure you remember its name and location.

Once you’ve created or selected your key pair, click Launch Instances. Your instance will spend a few minutes initialising before launching.

(e) Go to Services, select EC2, and click on Running instances to see the status of your new instance.

Install Docker on your instance:

Next, SSH into your instance to initialise it for use as your Dotscience runner. You will need:

  • Your key pair, downloaded from the AWS console. It has the name you assigned to it on creation and the file extension .pem.
  • The default user name for the machine image that you used to launch your instance. For an Ubuntu image, the user name is ubuntu.
  • The public DNS name of your runner, which you can copy and paste from the AWS EC2 Console: click on Running Instances to find it, where is is listed in the column Public DNS (IPv4).

1. SSH into your instance from the terminal using the information listed above. Use the syntax:

ssh -i <path/to/my-key-pair.pem> <user-name>@<public-dns-name>

2. If your instance isn’t new, you might want to uninstall any old Docker files that might be on there with the following command. If your VM is brand new, go to step 3.

sudo apt-get remove docker docker-engine docker.io

3. Install Docker

Use the installation script provided by docker.com

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

Add your user to the docker group so that you can execute Docker commands without using sudo

sudo usermod -a -G docker ubuntu

You need to log out and in again to the instance to activate the new user permissions. Do this by closing the terminal and opening a new one. Then ssh back into the instance with the command from step 1.

Check that Docker is up and running and doesn’t require sudo:

docker info

Create a Dotscience Runner on your VM:

  1. Go to the Runners section of Dotscience.
  2. Click on the Add Your Own Runner button.
  3. Select the Runner type, Storage space and Units that are most appropriate to your machine. Give the runner a name, and click Continue.
  4. Copy the commands given in the Connecting the runner section and run these on your new runner machine.
  5. Refresh the Dotscience Runners page. You should see your newly added runner is online.

GPU-enabled workflows

See the section on exposing GPU’s on your runner for more information on this.