Using Dotscience with arbitrary programs

If you have arbitrary code you need to run, you can use the ds command-line tool to integrate it with Dotscience.

This tutorial demonstrates how Dotscience can be used in Script mode. You can run any script (Python, R, bash, etc) on your workspace by installing the ds command-line tool on your Mac or Linux workstation, and using ds to interact with your Dotscience project and runners. For more information on the different modes in which Dotscience can be used, see the reference section on Dotscience modes.

Install the Dotscience (ds) client

sudo mkdir -p /usr/local/bin
sudo curl -sSL -o /usr/local/bin/ds$(uname -s)/ds

Make the binary executable

sudo chmod +x /usr/local/bin/ds

If you’re running on a custom install of Dotscience, you’ll need to set the server URL, for example if your hub is then run:

ds set server-url

Make sure to use your server address!

Log into Dotscience and go to Account > Keys and copy your API key.

echo <api key> | ds login <username>

Configure the (ds) client

By default the ds is configured to speak to If you are a user of our SaaS service you don’t need to configure ds.

Note: If you have an enterprise installation of Dotscience you would need to set the location of the Dotscience hub with

ds set server-url http(s)://your-dotscience-hub-url

A hello world run

Create a directory to run our tutorial

mkdir ds-run-test
cd ds-run-test

Create a project in Dotscience with ds

$ ds project create hello-ds

Run a sample program - the first argument is the local directory which get copied into the workspace, the second argument is the workspace name, the third argument is a docker image name and then the command you want to run

echo 'import dotscience as ds; ds.script(); ds.start(); ds.publish("hello, world")' >
ds run -p hello-ds --upload-path . python

Executing run ID 60691107-51b7-4d51-9e23-a0a972378161...
    "description": "hello, world",
    "end": "20191111T151011.932338",
    "input": [],
    "labels": {},
    "output": [],
    "parameters": {},
    "start": "20191111T151011.932327",
    "summary": {},
    "version": "1",
    "workload-file": ""
Task succeeded.
Submitted by userxxx.
Type: command
Image: ('[email protected]:bab2a7f55599a6222690c318a633b71cb65018fb7c5b8d78c3630ef8bcfb5dd6')
Command: "python" ""
Ran from 2019-11-11T15:10:10.319691039Z until 2019-11-11T15:10:17.52391425Z

Use the Dotscience Python Library to annotate your script with input, output, params & stats, and runs will start appearing in the Runs tab.

The ds run we did above registered the following Dotscience run

The compute for this is provided by the runner and this can be either your own self-service compute or one that is available on Dotscience. You can run any scripts with ds run and Dotscience has an internal scheduler that sends the script to be run on the first available runner.