Run Dotscience projects anywhere
Dotscience has the concept of runners - these are machines which are spun up to handle a task. Runners can be managed by dotscience or your own hardware. When tasks are started runners will automatically synchronise filesystems from dotscience and periodically update Dotscience with new commits so that you can analyse your work as you go.
Every new account will auto-provision a new runner, which will be shut down after 2 hours of inactivity. When you create a new task (i.e start Jupyter, run a python script or build a new model), runners will auto provision again, ready to be used.
To provision a new runner, go to the runners section of dotscience and click “Add managed runner” (notice there is already one in the list which is your default runner). You should see a window similar to this pop up:
Select the options which are appropriate to you and press continue - note that your default runner will not be GPU enabled, so it might be appropriate to make this one GPU so that you have both types available. You will be taken to the information about the runner
In the top right hand corner you should see the status of the runner - while the runner is spinning up (or shutting down) this will show offline, and may take a couple of minutes to be ready. If you’ve created your own runner using your own hardware, this will show offline until you run the docker command.
To use this runner, let’s head back to projects and select a project or create a new one. Click “Settings” and you should see the following:
The “Change runner” drop down provides you with a number of options
- Any available runner - tasks will randomly select a runner on which to launch.
- CPU - tasks will only launch on CPU runners
- GPU - NVIDIA - tasks will only launch on GPU nvidia-runtime enabled runners
- GPU - NVIDIA (legacy) - as above, but for the legacy NVIDIA runtime.
- managed-runner/gpu-runner - tasks will only ever launch on the named runner.
If you’re mid-project and haven’t yet set up a runner, or your runner was scaled down due to inactivity, you will be prompted to spin it up again:
Which will take you to:
Adding your own hardware
You can also spin up runners which work on your own machines or VMs. You can do this by clicking “add your own machine” from the project settings page as shown above, or from runners -> Add a runner. You will be able to fill in the same form where you can specify how much space you wish to take up and whether your machine is GPU-enabled.
When you click “continue” you will be taken to information about your runner again, but will need to run the docker command provided on the machine in order to use it.
To do this, click “copy to clipboard” and then, from a terminal or SSH to the machine, paste it and press enter. This should spin up a docker container, at which point the status in the top right of your runner view should show “online”.