Use Dotscience from the command line

Hi there! Welcome to the user manual for the ds command-line tool, which lets you drive Dotscience from your shell or automate it using scripting.

Installing `ds`

To install the latest ds cli, run the following:

sudo curl -sSL -o /usr/local/bin/ds \
  https://get.dotmesh.io/$(uname -s)/ds

# Make the client binary executable
sudo chmod +x /usr/local/bin/ds

Introduction and Concepts.

The Dotscience Service is the central hub that controls Dotscience activities in your organisation. It acts as a manager and a store for the canonical versions of your code and data, and stores a history of your activities.

Dotscience Runners are the computers that actually run your workloads. Your administrator sets them up, and they connect to the Dotscience Service to ask for work to do and the data to do it on; and they submit results back.

The ds command-line tool is what you run on the computer you’re sat in front of, to control the Dotscience Service (and, indirectly, the Runners).

As such, the ds tool needs to know your Dotscience login details.

All commands begin with ds, then contain a command, which in turn may contain further subcommands, e.g.

ds
ds deployer
ds deployer create

Help for each command can be found by appending -h, --help, or referring to the below.

Authentication: `ds login USERNAME`.

If you’re running on a custom install of Dotscience, you’ll need to set the server URL, for example if your hub is https://hub-52-0-143-21.your.dotscience.net then run:

$ ds set server-url https://hub-52-0-143-21.your.dotscience.net

Make sure to use your server address!

Before running any of the interesting ds commands, you need to authenticate by running ds login with your username. It’ll prompt you for your password:

$ ds login alice
Password: Type your password here, it won't be echoed

Authenticated successfully.

Running a job: `ds run -p PROJECT [-d] [-I IMAGE] [OTHER OPTIONS...] COMMAND...`.

This command runs a job. It runs the given COMMAND in the Docker image called IMAGE, with the workspace of PROJECT mounted as the current working directory. If IMAGE is not given, then quay.io/dotmesh/dotscience-python3:latest, a copy of python:3 with the Dotscience Python library pre-installed, is used.

Unless you specify the -d option, the ds run command will reflect the output of your job back to your screen as you watch. If you hit Ctrl+C, it will terminate the job. At the end of the job, the resulting job metadata will be displayed on your screen. The use of -d will be explained below in the Asynchronous mode section.

There are several other options not listed above, which we will explore below:

Uploading local files to the project workspace: `-u DIRECTORY`.

If you are editing your scripts locally, you can ask ds run to upload them into your project workspace for you, so you don’t have to upload them manually through the Dotscience web interface. This is accomplished with the -u DIRECTORY flag, which causes all files in the directory named by DIRECTORY (or its subdirectories) to be uploaded into the workspace before execution of the job. -u . uploads the current directory, which is often useful (and easy to type!)

By default, the entire contents of the directory DIRECTORY (including subdirectories) are synchronised into the workspace, so that it can be sent to the runner for execution and then committed to record the exact code that was executed in the audit trail - with the exception of any files whose names start with a . character, as by convention they are “hidden files”.

However, this can be overriden by using .dotscience-ignore files. If one of those is found in DIRECTORY or any of its subdirectories, then each line of the file (except blank lines or lines that start with #) is interpreted as a pattern. Any files matching that pattern in that directory or those beneath it will be ignored.

The patterns are Unix-style “globs”, which means that the following symbols in them have special meaning:

* matches any number of arbitrary characters
? matches any character
{A,B,...} matches any of the sub-patterns A, B, etc.
\X matches the character X exactly (even if it’s a special character such as *)
[a-zA-Z0-9\-_] matches any of the characters from a to z, or from A to Z, or from 0 to 9, or a hyphen (note the use of \- to mean an actual hyphen rather than indicating a range of characters), or _.
[!a-z] matches any character OTHER THAN those from a to z.

In addition, it’s possible to cancel an earlier ignore pattern by putting it in a .dotscience-ignore file, but prefixed with a - symbol. For instance, to enable uploading files whose names start with a ., simply put -.*

Specifying environment variables: `-e NAME` and `-e NAME=VALUE`.

You can specify shell environment variables to be in effect when running the command.

-e NAME=VALUE sets the environment variable NAME to the given VALUE.
-e NAME sets the environment variable NAME to whatever value the environment variable NAME has on the computer where you’re running ds, in effect passing it through.

Git integration: `-R REPO` and `-b REF`.

If the code or data you wish to work with originates in a Git repo, you can ask Dotscience to fetch it into your workspace before running the job. Pass the git URL (any URL you could pass to git clone) in as the REPO and it will be checked out into a subdirectory of the workspace, either named after the repo or called code if the system can’t deduce that from the repo URL.

By default, it will check out master; specifying a REF (which can be a branch name, a tag name, or an arbitrary commit hash) will cause it to check out that ref instead.

The first time you use this in a workspace, it will git clone the repo into your workspace. It’s actually saved into the workspace, so subsequent runs will find the checkout there; but it won’t be updated with subsequent changes in the git repo unless you specify -R and -b if appropriate.

If you have an SSH key associated with this project, then that ssh key will be available for git to use to access your repo using SSH ([email protected]:PATH style git URLs). See Managing Secrets below.

Specifying runner labels: `-r NAME` and `-r NAME=VALUE`.

By default, the job will be run by the first free runner that picks it up. However, by passing runner labels through, you can make sure that only runners satisfying some requirement will try to run your job, or provide configuration for the runner that picks it up. The meanings of the NAMEs and VALUEs depends on the runners your Dotscience account is configured with - consult your administrator for details.

-r NAME=VALUE sets the runner label NAME to the given VALUE.
-r NAME sets the runner lab NAME to whatever value the environment variable NAME has on the computer where you’re running ds.

Asynchronous mode: `-d`.

If you pass the -d flag, the ds run will output a task ID once execution has been scheduled, then return you to your prompt rather than waiting for the job to actually run.

This task ID can then be used with the commands listed in the next section, Task Management Commands.

Task Management Commands.

Jobs started with ds run - be they asynchronous (-d) or not - as well as JupyterLab instances started through the Dotscience web interface all count as “tasks”, and can be managed with ds commands.

Listing tasks: `ds task ls`.

The ds task ls command lists all the tasks known to your account. The columns in the output are ID (the task ID), STATUS (whether it’s running or in some other state), WORKLOAD (jupyter for JupyterLab or command for ds run), and AGE (when the task was created).

Note that terminated tasks remain in the list for several days before being removed, unless manually removed with ds task rm.

Examining task details: `ds task inspect ID`.

ds task inspect ID returns all the details of a task, as JSON.

Terminating a task: `ds task stop ID`.

ds task stop ID requests that a running task be stopped. It might take a while for the task to finish cleanly - any pending changes need to be uploaded back to the Hub, so watch the task’s status to find out when it’s completed!

Deleting a task: `ds task rm ID`.

ds task rm ID removes a terminated task from the list. It’s only really useful if you want to clean up the list, when you’re sure you won’t want any further information about the task.

Getting task logs: `ds logs [-v] [-f] ID`.

ds logs ID prints out all the stored logs for the task.

If -f is given, then once it has printed out all stored logs, if the task is still running, it will continue to print out logs as they happen, until the task terminates.

If -v is given, then all log entries will be printed; otherwise, only log output from your workload will be printed. -v mode is mainly useful for diagnosing infrastructure issues. In -v mode, every log entry is prefixed with a header consisting of the timestamp, the log entry type (omitted for workload standard output), a colon, then the line; without -v, workload standard output lines are emitted as-is, while workload standard error lines have ERROR: prepended to them.

Managing projects.

Getting a list of projects: `ds project ls [-q]`.

ds project ls lists all the projects in your account, including their ID and name. It’s quite possible to have multiple projects with the same name - if a project is in your account because it was shared with you and then you made a fork of it, for instance, you will have exactly that situation! Therefore, the IDs are often necessary to precisely specify which project you are referring to in commands that take a project name or ID.

This view will also show the datasets which are associated with this project and on what path they are mounted.

If you pass the -q flag, then rather than a table listing the project ID, NAME, workspace DOTS, COLLABORATORS and AGE columns, just the project names are listed (with no table heading).

Creating a project: `ds project create`.

This command creates a new project, and prints its ID out the console.

Describe a project: `ds project inspect PROJECT`.

ds project inspect prints out the full details of a project, in JSON format.

Deleting a project: `ds project rm PROJECT`.

This command deletes a project. Don’t run it by accident, as there’s no undelete!

List files in a project: `ds file ls [-q] PROJECT`.

To list the files in a project’s workspace, use ds file ls PROJECT. The result is a table with the file’s NAME, SIZE, and LAST MODIFIED timestamp; but if you specify -q it just lists the names.

Link a dataset to a project: `ds project link PROJECT DATASET PATH`

This command links a dataset to a project on PATH, so that when tasks are launched, the dataset is mounted at ./<PATH>. For example if you were to use the path s3 the path to accessing the contents of the dataset from the task would begin ./s3/.

Unlink a dataset from a project: `ds project unlink PROJECT DATASET`

This command unlinks a dataset from a project so that it no longer shows up as a mount when tasks are launched.

Managing datasets

Getting a list of datasets: `ds dataset ls [-q]`.

ds dataset ls lists all the projects in your account, including their ID and name. It will also show the sync status of the dataset, meaning - if it is an aws s3 dataset - how much of the s3 dataset has been downloaded to Dotscience.

If you pass the -q flag, then rather than a table listing the dataset ID, NAME, etc columns, just the dataset names are listed (with no table heading).

Creating a dataset: `ds dataset create s3 --name NAME --access-key-id AWS_ACCESS_KEY --secret-access-key AWS_SECRET_ACCESS_KEY --bucket BUCKET_NAME`.

This command creates a new s3 dataset and prints the ID to the console.

Describe a dataset: `ds dataset inspect DATASET`.

ds dataset inspect prints out the full details of a dataset, in JSON format.

Deleting a dataset: `ds dataset rm DATASET`.

This command deletes a dataset. Don’t run it by accident, as there’s no undelete!

Managing runs.

Run listings: `ds runs ls --summary-type=SUMMARY PROJECT`.

You can get a list of runs that have happened in a project with ds runs ls PROJECT (the project name or ID both work, and you’ll need to use the ID if you have multiple projects with the same name). Only runs with a summary statistic of type SUMMARY are listed, and the value of that statistic is shown for every run in the SCORE column; if you omit --summary-type the system will pick one arbitrarily, which is just what you want if you only have one summary statistic in your project.

Details of a run: `ds runs inspect --project PROJECT RUN`.

You can view the full details of a run in a given project with ds runs inspect. The output is JSON, in the Run Metadata Format.

Managing runners.

List runners: `ds runner ls [-q]`.

This command gets a list of all the runners attached to your account. The output is a table listing the ID, NAME, how many RUNNING TASKS it has, its STATUS, its TYPE (CPU or GPU), and its AGE; but if you specify -q then just the names are listed, with no table heading.

Details of a runner: `ds runner inspect RUNNER`.

ds runner inspect RUNNER returns the full details of the runner in JSON format, including a list of tasks the runner has run or is still running.

Create a runner: `ds runner create [-q] [-d DESCRIPTION] [-s SIZE] [-t cpu|gpu-nvidia-runtime] [NAME]`.

This command creates a new runner.

All the fields are optional: if no NAME is given, one will be picked. If no -t type is specified, cpu will be assumed. If no SIZE is given for the runner’s storage size in gigabytes, 10 will be assumed.

The output will look something like this:

Runner ID:        c871506a-93bb-49df-9ba7-6dca75a476bf
Runner name:      791d4f4a-untitled
Runner type:    cpu
Runner API token: PLBMTXVP3RLLTUFOEN3FOBCYDSPHL7BSLDZXVGXX3AAGZTZBBLJQ====

To start a runner, run:

docker pull quay.io/dotmesh/dotscience-runner:0.5.0 && \
docker run --name dotscience-runner -d -e TOKEN=PLBMTXVP3RLLTUFOEN3FOBCYDSPHL7BSLDZXVGXX3AAGZTZBBLJQ==== \
--restart always -v /var/run/docker.sock:/var/run/docker.sock \
-v dotscience-task-spool:/spool \
quay.io/dotmesh/dotscience-runner:0.5.0

The latter part is the command to run that will bring the runner up on any Docker installation. If you use the -q flag, then that’s all that is output; the metadata and instructional text is suppressed.

Remove a runner: `ds runner rm RUNNER`.

This command gets rid of a runner.

Managing secrets.

List secrets: `ds secret ls [-q]`.

This will list all the secrets known to your account. It outputs a table listing the ID, NAME, TYPE, PROJECT and AGE; unless you specify -q, in which case you just get the names.

List details of a secret: `ds secret inspect SECRET`.

This prints out the details of a secret, in JSON form. The data field, for an SSH key, will be base64-encoded JSON containing public_key and private_key fields; they can be extracted using jq like so to get values that can be used with a normal ssh client:

$ ds secret inspect SECRET | jq -r .[0].data | base64 -d | jq -r .public_key | base64 -d
$ ds secret inspect SECRET | jq -r .[0].data | base64 -d | jq -r .private_key | base64 -d

Generate a secret: `ds secret generate --project PROJECT --name NAME`.

This will generate an SSH keypair associated with PROJECT, called NAME, and will output the public part of the keypair.

Removing a secret: `ds secret rm SECRET`.

You can delete a secret from your account with ds secret rm SECRET.

Other Useful Commands.

These aren’t commands you’ll need very often, but here they are for completeness!

Online help: `ds help`.

You can obtain online help on a command by typing ds help COMMAND or ds COMMAND --help:

$ ds help version
Display the version of the Dotscience command-line tool

Usage:
  ds version [flags]

Flags:
  -h, --help   help for version

Checking the version of the `ds` tool: `ds version`.

$ ds version
Dotscience command-line tool version: 0.1

Connecting to different Dotscience hubs: `ds set server-url`.

By default, ds connects to the Dotscience SaaS server, but if you have a dedicated instance of your own, use ds set server-url to tell ds to connect to it. Set it to the scheme and domain part of the base URL you use for logging into Dotscience, such as https://cloud.dotscience.com; do not include anything after the domain name!

Complete reference

This reference is up to date for the Dotscience command-line tool as of April 6th 2020

ds

Dotscience is a platform for managing, running,
and tracking data science workloads.

See https://www.dotscience.com/ for more details.

The ds tool provides access to Dotscience functionality
from your command line.

Usage:
  ds [command]

Available Commands:
  addon         Manage and view addons
  ci            Manage ci configurations - create, list and modify ci configurations
  dataset       Manage datasets - create, list and modify datasets
  deployer      Manage deployers - create, list and modify deployers
  deployment    Manage deployment - create, list and modify deployments
  file          View files
  help          Help about any command
  login         Login and save credentials
  logs          Task logs
  model         Manage models - list and build models
  project       Manage projects - create, list and modify projects
  rebuild-cache Rebuild the cache for the specified project, or if not specified, all projects on the Hub
  rp            Manage runner profiles - create, list and modify runners (requires cluster admin permissions)
  run           Run a Dotscience task
  runner        Manage runners - create, list and modify runners
  runs          View project runs
  secret        Manage secrets - create and list SSH keys
  set           Update configuration settings
  task          Manage tasks - control tasks
  version       Display the version of the Dotscience command-line tool

Flags:
  -h, --help   help for ds

Use "ds [command] --help" for more information about a command.

ds addon

Manage and view addons

Usage:
  ds addon [flags]
  ds addon [command]

Available Commands:
  disable     Disable addon for a task - meltano, tensorboard
  enable      Enable addon for a task - meltano, tensorboard

Flags:
  -h, --help   help for addon

Use "ds addon [command] --help" for more information about a command.

ds addon disable

Disable addon for a task - meltano, tensorboard

Usage:
  ds addon disable [flags]
  ds addon disable [command]

Available Commands:
  meltano     Disable Meltano addon for the running task
  tensorboard Disable Tensorboard addon for the running task

Flags:
  -h, --help   help for disable

Use "ds addon disable [command] --help" for more information about a command.

ds addon disable meltano

Disable Meltano addon for the running task

Usage:
  ds addon disable meltano [OPTIONS] [flags]

Flags:
  -h, --help   help for meltano

ds addon disable tensorboard

Disable Tensorboard addon for the running task

Usage:
  ds addon disable tensorboard [OPTIONS] [flags]

Flags:
  -h, --help   help for tensorboard

ds addon enable

Enable addon for a task - meltano, tensorboard

Usage:
  ds addon enable [flags]
  ds addon enable [command]

Available Commands:
  meltano     Enable Meltano addon for the running task
  tensorboard Enable Tensorboard addon for the running task

Flags:
  -h, --help   help for enable

Use "ds addon enable [command] --help" for more information about a command.

ds addon enable meltano

Enable Meltano addon for the running task

Usage:
  ds addon enable meltano [OPTIONS] [flags]

Flags:
  -h, --help   help for meltano

ds addon enable tensorboard

Enable Tensorboard addon for the running task

Usage:
  ds addon enable tensorboard [OPTIONS] [flags]

Flags:
  -h, --help   help for tensorboard

ds ci

Manage ci configurations - create, list and modify ci configurations

Usage:
  ds ci [flags]
  ds ci [command]

Available Commands:
  create      Create ci configurations - gitlab
  inspect     Display detailed information on one or more ci configurations
  ls          List ci configurations
  rm          Remove one or more ci configurations

Flags:
  -h, --help   help for ci

Use "ds ci [command] --help" for more information about a command.

ds ci create

Create ci configurations - gitlab

Usage:
  ds ci create [flags]
  ds ci create [command]

Available Commands:
  gitlab      Create a new Gitlab CI configuration

Flags:
  -h, --help   help for create

Use "ds ci create [command] --help" for more information about a command.

ds ci create gitlab

Create a new Gitlab CI configuration

Usage:
  ds ci create gitlab [OPTIONS] [flags]

Flags:
      --default-builder        Set this configuration as the default builder for your account. Default false
      --description string     Description of config
  -h, --help                   help for gitlab
      --project string         name of the gitlab project
      --project-id string      ID of the gitlab project
      --ref string             git ref to run this on, e.g master
      --token string           gitlab token to use for authentication
      --trigger-token string   token used to trigger the pipeline
      --url string             gitlab instance URL

ds ci inspect

Display detailed information on one or more ci configurations

Usage:
  ds ci inspect [OPTIONS] CONFIGURATION [CONFIGURATION...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds ci ls

List ci configurations

Usage:
  ds ci ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds ci rm

Remove one or more ci configuration.

Usage:
  ds ci rm [OPTIONS] CI_CONFIG [CI_CONFIG...] [flags]

Aliases:
  rm, remove

Examples:

$ ds ci rm prodConfig
prodConfig

Flags:
  -h, --help   help for rm

ds dataset

Manage datasets - create, list and modify datasets

Usage:
  ds dataset [flags]
  ds dataset [command]

Available Commands:
  create      Create datasets - s3
  inspect     Display detailed information on one or more datasets
  ls          List datasets
  rm          Remove one or more datasets

Flags:
  -h, --help   help for dataset

Use "ds dataset [command] --help" for more information about a command.

ds dataset create

Create datasets - s3

Usage:
  ds dataset create [flags]
  ds dataset create [command]

Available Commands:
  s3          Create a new s3 dataset

Flags:
  -h, --help   help for create

Use "ds dataset create [command] --help" for more information about a command.

ds dataset create s3

Create a new s3 dataset

Usage:
  ds dataset create s3 [OPTIONS] [flags]

Flags:
      --access-key-id string       AWS programmatic access key ID - account must have access to the bucket
      --bucket string              Name of s3 bucket
  -h, --help                       help for s3
      --secret-access-key string   AWS programmatic secret access key - must be a matching pair with access key ID and have access to the bucket

ds dataset inspect

Display detailed information on one or more datasets

Usage:
  ds dataset inspect [OPTIONS] DATASET [DATASET...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds dataset ls

List datasets

Usage:
  ds dataset ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds dataset rm

Remove one or more dataset.

Usage:
  ds dataset rm [OPTIONS] DATASET [DATASET...] [flags]

Aliases:
  rm, remove

Examples:

$ ds dataset rm prodDataset
prodDataset

Flags:
  -h, --help   help for rm

ds deployer

Manage deployers - create, list and modify deployers

Usage:
  ds deployer [flags]
  ds deployer [command]

Available Commands:
  create      Create a new deployer
  inspect     Display detailed information on one or more deployers
  ls          List deployers
  rm          Remove one or more deployers

Flags:
  -h, --help   help for deployer

Use "ds deployer [command] --help" for more information about a command.

ds deployer create

Create a new deployer

Usage:
  ds deployer create [OPTIONS] [flags]

Flags:
      --custom-hostnames     Whether to allow custom deployment hostnames for this deployer
  -d, --description string   Description
  -h, --help                 help for create
      --managed              Managed deployer (shared with other cluster users)
      --max-replicas int     Max number of replicas allowed for the deployment (default 2)
      --namespace string     Deployment namespace (default "default")
  -q, --quiet                Quiet mode (just output the runner startup command)
      --subdomain string     Subdomain to use for new deployments (for example: '.app.cloud.dotscience.net')
  -t, --token string         Deployer token (optional)

ds deployer inspect

Display detailed information on one or more deployers

Usage:
  ds deployer inspect [OPTIONS] DEPLOYER [DEPLOYER...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds deployer ls

List deployers

Usage:
  ds deployer ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds deployer rm

Remove one or more deployers.

Usage:
  ds deployer rm [OPTIONS] DEPLOYER [DEPLOYER...] [flags]

Aliases:
  rm, remove

Examples:

$ ds deployer rm prodDeployer
prodDeployer

Flags:
  -h, --help   help for rm

ds deployment

Manage deployment - create, list and modify deployments

Usage:
  ds deployment [flags]
  ds deployment [command]

Available Commands:
  create      Create a new deployment
  inspect     Display detailed information on one or more deployments
  ls          List deployments
  rm          Remove one or more deployments

Flags:
  -h, --help   help for deployment

Use "ds deployment [command] --help" for more information about a command.

ds deployment create

Create a new deployment

Usage:
  ds deployment create [OPTIONS] [flags]

Flags:
  -d, --deployer string        Deployer to use
  -h, --help                   help for create
      --host string            Hostname for ingress controller
  -i, --image-name string      Docker image name
      --ingress-class string   Ingress class to use (default "webrelay")
      --model-classes string   Path to model classes file
      --model-name string      Model name
      --namespace string       Namespace
  -p, --port int               Docker container port (default 8501)
  -q, --quiet                  Quiet mode (just output the runner startup command)
  -r, --replicas int           Replicas to run (number of containers to start) (default 1)

ds deployment inspect

Display detailed information on one or more deployments

Usage:
  ds deployment inspect [OPTIONS] DEPLOYMENT [DEPLOYMENT...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds deployment ls

List deployments

Usage:
  ds deployment ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
  -d, --deployer string   Deployer ID or name
      --format string     Pretty-print using a Go template
  -h, --help              help for ls
  -q, --quiet             Only display names

ds deployment logs

Error: "ds deployment logs" requires 1 argument(s).
See 'ds deployment logs --help'.

Usage:  ds deployment logs [OPTIONS] DEPLOYMENT [flags]

Display deployment logs
Usage:
  ds deployment logs [OPTIONS] DEPLOYMENT [flags]

Example:

$ ds deployment logs 3712682c-a201-491a-84db-595871d3d86e
2020-04-06 12:36:55.790737: I tensorflow_serving/model_servers/server.cc:86] Building single TensorFlow model file config:  model_name: model model_base_path: /models/model
2020-04-06 12:36:55.791019: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2020-04-06 12:36:55.791039: I tensorflow_serving/model_servers/server_core.cc:573]  (Re-)adding model: model
2020-04-06 12:36:55.891407: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: model version: 1}
2020-04-06 12:36:56.035047: I tensorflow_serving/model_servers/server.cc:378] Exporting HTTP/REST API at:localhost:8501 ...
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...

ds deployment rm

Remove one or more deployments.

Usage:
  ds deployment rm [OPTIONS] DEPLOYMENT [DEPLOYMENT...] [flags]

Aliases:
  rm, remove

Examples:

$ ds deployment rm model-staging
odel-staging

Flags:
  -h, --help   help for rm

ds file

View files

Usage:
  ds file [flags]
  ds file [command]

Available Commands:
  download    Download files from a project
  ls          List files for a project
  upload      Upload file to a project

Flags:
  -h, --help   help for file

Use "ds file [command] --help" for more information about a command.

ds file download

Download files from a project

Usage:
  ds file download [OPTIONS] PATH DESTINATION [flags]

Aliases:
  download, download

Flags:
  -c, --commit string        Commit ID (default "latest")
  -d, --destination string   Destination where to download the file or directory
  -f, --force                Overwrite destination
  -h, --help                 help for download
      --path string          File or directory to download
  -p, --project string       Project ID or name

ds file ls

List files for a project

Usage:
  ds file ls [OPTIONS] PROJECT [flags]

Aliases:
  ls, list

Flags:
      --format string    Pretty-print using a Go template
  -h, --help             help for ls
  -p, --project string   Project ID or name
  -q, --quiet            Only display names

ds file upload

Upload file to a project

Usage:
  ds file upload [OPTIONS] FILEPATH [flags]

Aliases:
  upload, upload

Flags:
  -f, --filename string   Filename
  -h, --help              help for upload
      --path string       Override filename (for example if you want to upload file to a specific directory)
  -p, --project string    Project ID or name

ds help

Help provides help for any command in the application.
Simply type ds help [path to command] for full details.

Usage:
  ds help [command] [flags]

Flags:
  -h, --help   help for help

Login and save credentials

Usage:
  ds login [OPTIONS] [flags]

Flags:
  -h, --help              help for login
  -p, --password string   Password
  -u, --username string   Username.

ds logs

Task logs

Usage:
  ds logs [OPTIONS] TASK [flags]

Flags:
  -f, --follow          Follow logs output
      --format string   Pretty-print using a Go template
  -h, --help            help for logs
  -q, --quiet           Only display names
      --task string     Task ID
  -v, --verbose         Display verbose logs

ds model

Manage models - list and build models

Usage:
  ds model [flags]
  ds model [command]

Available Commands:
  build       Build model image
  builds      Build list for a model
  inspect     Display detailed information on one or more models
  ls          List models

Flags:
  -h, --help   help for model

Use "ds model [command] --help" for more information about a command.

## ds model ls
Lists all available models for a logged in user.

Example

ds model ls
ID                                     NAME                FRAMEWORK           BUILD STATUS                                                                                                                 AGE
20fe31f5-7df6-4fc4-9ad1-1d70bc146d1a   roadsigns           tensorflow-model    completed (quay.io/dotscience-playground/models:77da5825-a4c4-41b2-95eb-34280c4c688a-12478713-27c4-4bbe-985d-8ce69a2199c5)   5 days
d71e35b8-9cfd-492b-a708-8a825ab5e639   mnist-epochs-1      tensorflow-model    completed (quay.io/dotscience-playground/models:77da5825-a4c4-41b2-95eb-34280c4c688a-acf569a3-e355-48ad-8622-901d2f5629da)   13 days

ds model build

Builds a model into a docker image. Supported model frameworks are tensorflow and scikit-learn.

Usage:  ds model build [OPTIONS] MODEL  [flags]

Build model image
Usage:
  ds model build [OPTIONS] MODEL  [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for build

ds model builds

List of builds for a given model.

Usage:  ds model builds [OPTIONS] MODEL  [flags]

Build list for a model
Usage:
  ds model builds [OPTIONS] MODEL  [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for builds
  -q, --quiet           Only display IDs

ds project

Manage projects - create, list and modify projects

Usage:
  ds project [flags]
  ds project [command]

Available Commands:
  create      Create a new project
  inspect     Display detailed information on one or more projects
  link        link a project to a dataset on a given path
  link        unlink a dataset from a project
  ls          List projects
  rm          Remove one or more projects

Flags:
  -h, --help   help for project

Use "ds project [command] --help" for more information about a command.

ds project create

Create a new project

Usage:
  ds project create [OPTIONS] [flags]

Flags:
  -h, --help   help for create

ds project inspect

Display detailed information on one or more projects

Usage:
  ds project inspect [OPTIONS] PROJECT [PROJECT...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds project link

link a project to a dataset on a given path

Usage:
  ds project link ID DATASET PATH [flags]

Flags:
-h, --help   help for link

ds project ls

List projects

Usage:
  ds project ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds project rm

Remove one or more projects.

Usage:
  ds project rm [OPTIONS] PROJECT [PROJECT...] [flags]

Aliases:
  rm, remove

Examples:

$ ds project rm my-project
my-project

Flags:
  -h, --help   help for rm

ds rp

Manage runner profiles - create, list and modify runners (requires cluster admin permissions)

Usage:
  ds rp [flags]
  ds rp [command]

Available Commands:
  create      Create a new runner profile
  inspect     Display detailed information on one or more runner profiles
  ls          List runner profiles
  rm          Remove one or more runner profiles
  update      Update an existing runner profile

Flags:
  -h, --help   help for rp

Use "ds rp [command] --help" for more information about a command.

ds rp create

Create a new runner profile

Usage:
  ds rp create [OPTIONS] [flags]

Flags:
  -d, --description string            Description
      --disk-size int                 Disk size (default 200)
      --gpu-count int                 GPU count
      --gpu-type string               GPU type (if any), you can also choose machine size that includes GPU
  -h, --help                          help for create
      --machine-image string          Machine image (AMI) - optional
      --machine-image-family string   Machine image family (AMI), used by GCP
      --preemtible                    Preemtible (if supported by the platform)
      --pricing string                Optional pricing info for the users to see
  -r, --runtime string                Runner runtime type (cpu, gpu-nvidia-runtime or gpu-nvidia-docker) (default "cpu")
      --size string                   Machine size (for example 'p3.2xlarge' on AWS)

ds rp inspect

Display detailed information on one or more runner profiles

Usage:
  ds rp inspect [OPTIONS] PROFILE [PROFILE...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds rp ls

List runner profiles

Usage:
  ds rp ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds rp rm

Remove one or more runner profiles.

Usage:
  ds rp rm [OPTIONS] PROFILE [PROFILE...] [flags]

Aliases:
  rm, remove

Examples:

$ ds rp rm gpu-profile
gpu-profile

Flags:
  -h, --help   help for rm

ds rp update

Update an existing runner profile

Usage:
  ds rp update [OPTIONS] [flags]

Flags:
  -d, --description string            Description
      --disk-size int                 Disk size (default 200)
      --gpu-count int                 GPU count
      --gpu-type string               GPU type (if any), you can also choose machine size that includes GPU
  -h, --help                          help for update
      --machine-image string          Machine image (AMI) - optional
      --machine-image-family string   Machine image family (AMI), used by GCP
      --preemtible                    Preemtible (if supported by the platform)
      --pricing string                Optional pricing info for the users to see
      --size string                   Machine size (for example 'p3.2xlarge' on AWS)

ds run

Run a Dotscience task

Usage:
  ds run [OPTIONS] COMMAND [COMMAND...] [flags]

Flags:
  -d, --async                    Run the command in the background.
  -c, --command stringArray      Command to run
  -e, --environment NAME=VALUE   Set a variable of the form NAME=VALUE in the shell environment when running the command
  -h, --help                     help for run
  -I, --image string             Docker image to use. If not set, will use the latest stable official dotscience python image
      --nvidia                   Use the nvidia GPU runtime
  -p, --project-name string      Project name to use
  -b, --ref git checkout         Branch, tag, or commit reference to start from in git (i.e a valid entry to git checkout). If this is not set and repo is set, will checkout master and pull the latest commit.
  -R, --repo string              Git repo to link to. If not set, will not work with git.
  -r, --runner KEY=VALUE         Specify KEY=VALUE labels to configure the runner.
  -S, --selector string          Selector to pick a specific runner, eg 'id=RUNNER ID'
  -u, --upload-path string       Upload the contents of the supplied directory while doing the run. If not supplied, will not do any uploads.
  -v, --verbose                  Display all internal system logging

ds runner

Manage runners - create, list and modify runners

Usage:
  ds runner [flags]
  ds runner [command]

Available Commands:
  create              Create a new runner
  inspect             Display detailed information on one or more runners
  ls                  List runners
  rm                  Remove one or more runner
  show-startup-script Display the startup script for a runner

Flags:
  -h, --help   help for runner

Use "ds runner [command] --help" for more information about a command.

ds runner create

Create a new runner

Usage:
  ds runner create [OPTIONS] [flags]

Flags:
  -d, --description string      Description
  -h, --help                    help for create
  -s, --pool-size int           Pool size (GB) (default 10)
  -q, --quiet                   Quiet mode (just output the runner startup command)
  -t, --runner-runtime string   Runner runtime type, defaults to 'cpu' (cpu or gpu-nvidia-runtime for GPU runners) (default "cpu")

ds runner inspect

Display detailed information on one or more runners

Usage:
  ds runner inspect [OPTIONS] RUNNER [RUNNER...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds runner ls

List runners

Usage:
  ds runner ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds runner rm

Remove one or more runners.

Usage:
  ds runner rm [OPTIONS] RUNNER [RUNNER...] [flags]

Aliases:
  rm, remove

Examples:

$ ds runner rm my-runner
my-runner

Flags:
  -h, --help   help for rm

ds runner show-startup-script

Display the startup script for a runner

Usage:
  ds runner show-startup-script [OPTIONS] RUNNER [flags]

Flags:
  -h, --help   help for show-startup-script

ds runs

View project runs

Usage:
  ds runs [flags]
  ds runs [command]

Available Commands:
  inspect     Display detailed information on one or more project runs
  ls          List runs for a project

Flags:
  -h, --help   help for runs

Use "ds runs [command] --help" for more information about a command.

ds runs inspect

Display detailed information on one or more project runs

Usage:
  ds runs inspect [OPTIONS] RUN [RUN...] [flags]

Flags:
  -f, --format string    Format the output using the given Go template
  -h, --help             help for inspect
      --project string   Project ID or name

ds runs ls

List runs for a project

Usage:
  ds runs ls [OPTIONS] PROJECT [flags]

Aliases:
  ls, list

Flags:
      --format string         Pretty-print using a Go template
  -h, --help                  help for ls
      --project string        Project ID or name
  -q, --quiet                 Only display names
      --summary-type string   Summary type
  -v, --verbose               Verbose - display more information

ds secret

Manage secrets - create and list SSH keys

Usage:
  ds secret [flags]
  ds secret [command]

Available Commands:
  generate    Generate a public and private key SSH pair
  inspect     Display detailed information on one or more secrets
  ls          List secrets
  rm          Remove one or more secrets

Flags:
  -h, --help   help for secret

Use "ds secret [command] --help" for more information about a command.

ds secret generate

Generate a public and private key SSH pair

Usage:
  ds secret generate [OPTIONS] [flags]

Flags:
  -h, --help             help for generate
      --name string      Secret name
  -p, --project string   Project ID

ds secret inspect

Display detailed information on one or more secrets

Usage:
  ds secret inspect [OPTIONS] SECRET [SECRET...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds secret ls

List secrets

Usage:
  ds secret ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds secret rm

Remove one or more secrets.

Usage:
  ds secret rm [OPTIONS] SECRET [SECRET...] [flags]

Aliases:
  rm, remove

Examples:

$ ds secret rm secret-name-or-id
secret-name-or-id

Flags:
  -h, --help   help for rm

ds set

Update configuration settings

Usage:
  ds set [command]

Available Commands:
  agent-image       Update the Dotscience Agent docker image
  default-namespace Update the default namespace
  server-url        Update the Dotscience Server URL

Flags:
  -h, --help   help for set

Use "ds set [command] --help" for more information about a command.

ds set agent-image

Update the Dotscience Agent docker image

Usage:
  ds set agent-image <docker image> [flags]

Flags:
  -h, --help   help for agent-image

ds set default-namespace

Update the default namespace

Usage:
  ds set default-namespace <namespace> [flags]

Flags:
  -h, --help   help for default-namespace

ds set server-url

Update the Dotscience Server URL

Usage:
  ds set server-url <url> [flags]

Flags:
  -h, --help   help for server-url

ds task

Manage tasks - control tasks

Usage:
  ds task [flags]
  ds task [command]

Available Commands:
  inspect     Display detailed information on one or more tasks
  ls          List tasks for the runner
  rm          Remove one or more tasks
  stop        Stop (terminate) one or more tasks

Flags:
  -h, --help   help for task

Use "ds task [command] --help" for more information about a command.

ds task inspect

Display detailed information on one or more tasks

Usage:
  ds task inspect [OPTIONS] TASK [TASK...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds task ls

List tasks for the runner

Usage:
  ds task ls [OPTIONS] RUNNER [RUNNER...] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds task rm

Remove one or more tasks.

Usage:
  ds task rm [OPTIONS] TASK [TASK...] [flags]

Aliases:
  rm, remove

Examples:

$ ds task rm my-task-uuid
my-task-uuid

Flags:
  -h, --help   help for rm

ds task stop

Stop one or more tasks. Tasks cannot be restarted.

Usage:
  ds task stop [OPTIONS] TASK [TASK...] [flags]

Aliases:
  stop, remove

Examples:

$ ds task stop task-uuid
task-uuid

Flags:
  -h, --help   help for stop

ds version

Display the version of the Dotscience command-line tool

Usage:
  ds version [flags]

Flags:
  -h, --help   help for version

Installing ds

Introduction and Concepts.

Authentication: ds login USERNAME.

Running a job: ds run -p PROJECT [-d] [-I IMAGE] [OTHER OPTIONS...] COMMAND....

Uploading local files to the project workspace: -u DIRECTORY.

Specifying environment variables: -e NAME and -e NAME=VALUE.

Git integration: -R REPO and -b REF.

Specifying runner labels: -r NAME and -r NAME=VALUE.

Asynchronous mode: -d.