References

Dotscience Architecture

Dotscience components and their relationships

Overview

A Dotscience installation consists of two main components - the central Hub, and a pool of Runners.

Architecture Diagram

Physical components of a Dotscience installation

The Hub

The Dotscience Hub is the central repository of your workspace data and metadata, and provides the Web interface users log into to interact with Dotscience.

Actual workloads such as JupyterLab or command-line tasks submitted via ds run are not executed on the Hub, although the request to start them is always sent to the Hub; the Hub forwards those tasks to your Runners, tracks their state, and allows them to download and upload files from the workspace data stored on the Hub.

The Hub is also responsible for managing HTTP tunnels, so that you can view JupyterLab’s Web interface inside the Hub web application. The runners, where JupyterLab actually runs, may not have inbound Internet connectivity, so incoming connections to the JupyterLab Web interface are relayed via the Hub. The runners make only outbound connections to the hub.

The Runners

Runners are where your workloads are executed. They run a lightweight scheduler as a Docker container, which connects to the Hub and requests tasks to execute. Their storage is used for transient working state for your workloads, and as a cache of workspace data from the Hub.

Typical tasks would be JupyterLab, or a command-line task submitted via the ds run command.

To run a task, the runner performs the following steps:

  1. Ensure the required workspace data is on the runner and up-to-date. If this is the first task using a particular workspace or dataset, then it is copied from the Hub in its entirety; if it has been pulled onto this runner before, then only the changes need to be downloaded.

  2. Ensure the required Docker images for the workload are up-to-date on the runner (e.g., the supplied JupyterLab image, or your own Docker image for a command-line task).

  3. If required (JupyterLab only), set up a tunnel for incoming HTTP connections to the Hub to be relayed to the workload container.

  4. Start the workload in its own Docker container, with the appropriate workspace and any datasets mapped into its filesystem.

  5. Monitor the workload for new runs, and send them to the Hub (along with a snapshot of the generated data).

  6. When the workload finishes (e.g., Jupyter is stopped or the command-line task finishes), upload any final runs and output data, and clear up.

Each runner authenticates to the Hub using a unique runner token.

SaaS Architecture

Our SaaS offering, cloud.dotscience.com, provides a managed installation of the Hub, along with automatically provisioned managed Runners created in a public cloud.

However, users can also add their own Runners to gain access to additional processing power.

AWS Architecture

We also provide a CloudFormation template that can be customised and instantiated by the end-user, or accessed via Amazon’s AWS Marketplace, so that customers can create their own, separate, Dotscience installs in their own AWS accounts.

The CloudFormation template instantiates a Hub and a single Runner in EC2, with an S3 bucket used to communicate the runner token between the two. The Hub creates a new runner token at startup and places it in the bucket; the runner’s EC2 instance waits for the runner token to appear in the bucket, then uses it to start the Runner scheduling container.

The Hub’s persistent state is stored in a separate EBS volume, configured to not be deleted on termination, so that the Hub can be upgraded by instantiating a new EC2 instance from a newer AMI and just re-attach the existing volume.

The Runner’s persistent state is stored in its root partition, because it is just a cache and the Runner software can upgrade itself in-place.

The Runner and Hub are connected with a dedicated VPC, with an InternetGateway for external connectivity. The Runner and Hub each have their own SecurityGroup to control incoming connections. The instances are managed by AutoScaling Groups.

For more information, see also the AWS section of the security documentation section.

AWS Architecture Diagram

The following diagram was automatcially generated from the CloudFormation template.

Deployment in AWS