Tutorials

Script based development - with ds client

ds is a command-line tool, which lets you drive Dotscience from your shell or automate it using scripting.

This tutorial demonstrates how Dotscience can be used in Script mode. You can run any script (Python, R, bash, etc) on your workspace by installing the ds command-line tool on your Mac or Linux workstation, and using ds to interact with your Dotscience project and runners. For more information on the different modes in which Dotscience can be used, see the reference section on Dotscience modes.

Install the Dotscience (ds) client

sudo mkdir -p /usr/local/bin
sudo curl -sSL -o /usr/local/bin/ds https://get.dotmesh.io/$(uname -s)/ds

Make the binary executable

sudo chmod +x /usr/local/bin/ds

Log into Dotscience and go to Account > Keys and copy your API key.

echo <api key> | ds login <username>

Configure the (ds) client

By default the ds is configured to speak to https://cloud.dotscience.com. If you are a user of our SaaS service you don’t need to configure ds.

Note: If you have an entperprise installation of Dotscience you would need to set the location of the Dotscience hub with

ds set server-url http(s)://your-dotscience-hub-url

A hello world run

Create a directory to run our tutorial

mkdir ds-run-test
cd ds-run-test

Create a project in Dotscience with ds

$ ds project create hello-ds
2f6fc697-fd46-4ba9-a41f-05bcd569e36e

Run a sample program - the first argument is the local directory which get copied into the workspace, the second argument is the workspace name, the third argument is a docker image name and then the command you want to run

echo 'import dotscience as ds; ds.script(); ds.start(); ds.publish("hello, world")' > test.py
ds run -p hello-ds --upload-path . python test.py

Executing run ID 60691107-51b7-4d51-9e23-a0a972378161...
[[DOTSCIENCE-RUN:e4729f40-437a-45e1-bd70-e023df59961d]]{
    "description": "hello, world",
    "end": "20191111T151011.932338",
    "input": [],
    "labels": {},
    "output": [],
    "parameters": {},
    "start": "20191111T151011.932327",
    "summary": {},
    "version": "1",
    "workload-file": "test.py"
}[[/DOTSCIENCE-RUN:e4729f40-437a-45e1-bd70-e023df59961d]]
Task succeeded.
Submitted by userxxx.
Type: command
Image: quay.io/dotmesh/dotscience-python3:0.6.7 ('quay.io/dotmesh/[email protected]:bab2a7f55599a6222690c318a633b71cb65018fb7c5b8d78c3630ef8bcfb5dd6')
Command: "python" "test.py"
Ran from 2019-11-11T15:10:10.319691039Z until 2019-11-11T15:10:17.52391425Z
...

Use the Dotscience Python Library to annotate your script with input, output, params & stats, and runs will start appearing in the Runs tab.

The ds run we did above registered the following Dotscience run

The compute for this is provided by the runner and this can be either your own self-service compute or one that is available on Dotscience. You can run any scripts with ds run and Dotscience has an internal scheduler that sends the script to be run on the first available runner.