The ds command-line tool
Use Dotscience from the command line
Hi there! Welcome to the user manual for the ds
command-line tool,
which lets you drive Dotscience from your shell or automate it using
scripting.
Installing ds
To install the latest ds
cli, run the following:
sudo curl -sSL -o /usr/local/bin/ds \
https://get.dotmesh.io/$(uname -s)/ds
# Make the client binary executable
sudo chmod +x /usr/local/bin/ds
Introduction and Concepts.
The Dotscience Service is the central hub that controls Dotscience activities in your organisation. It acts as a manager and a store for the canonical versions of your code and data, and stores a history of your activities.
Dotscience Runners are the computers that actually run your workloads. Your administrator sets them up, and they connect to the Dotscience Service to ask for work to do and the data to do it on; and they submit results back.
The ds
command-line tool is what you run on the computer you’re sat
in front of, to control the Dotscience Service (and, indirectly, the
Runners).
As such, the ds
tool needs to know your Dotscience login details.
All commands begin with ds, then contain a command, which in turn may contain further subcommands, e.g.
ds
ds deployer
ds deployer create
Help for each command can be found by appending -h
, --help
, or referring to the below.
Authentication: ds login USERNAME
.
If you’re running on a custom install of Dotscience, you’ll need to set the server URL, for example if your hub is https://hub-52-0-143-21.your.dotscience.net
then run:
$ ds set server-url https://hub-52-0-143-21.your.dotscience.net
Make sure to use your server address!
Before running any of the interesting ds
commands, you need to authenticate by running ds login
with your username. It’ll prompt you for your password:
$ ds login alice
Password: Type your password here, it won't be echoed
Authenticated successfully.
Running a job: ds run -p PROJECT [-d] [-I IMAGE] [OTHER OPTIONS...] COMMAND...
.
This command runs a job. It runs the given COMMAND in the Docker image
called IMAGE, with the workspace of PROJECT mounted as the current
working directory. If IMAGE is not given, then
quay.io/dotmesh/dotscience-python3:latest
, a copy of python:3
with
the Dotscience Python library pre-installed, is used.
Unless you specify the -d
option, the ds run
command will reflect
the output of your job back to your screen as you watch. If you hit
Ctrl+C, it will terminate the job. At the end of the job, the
resulting job metadata will be displayed on your screen. The use of
-d
will be explained below in the Asynchronous mode section.
There are several other options not listed above, which we will explore below:
Uploading local files to the project workspace: -u DIRECTORY
.
If you are editing your scripts locally, you can ask ds run
to
upload them into your project workspace for you, so you don’t have to
upload them manually through the Dotscience web interface. This is
accomplished with the -u DIRECTORY
flag, which causes all files in
the directory named by DIRECTORY (or its subdirectories) to be
uploaded into the workspace before execution of the job. -u .
uploads the current directory, which is often useful (and easy to
type!)
By default, the entire contents of the directory DIRECTORY (including
subdirectories) are synchronised into the workspace, so that it can be
sent to the runner for execution and then committed to record the
exact code that was executed in the audit trail - with the exception
of any files whose names start with a .
character, as by convention
they are “hidden files”.
However, this can be overriden by using .dotscience-ignore
files. If
one of those is found in DIRECTORY or any of its subdirectories, then
each line of the file (except blank lines or lines that start with
#
) is interpreted as a pattern. Any files matching that pattern in
that directory or those beneath it will be ignored.
The patterns are Unix-style “globs”, which means that the following symbols in them have special meaning:
*
matches any number of arbitrary characters?
matches any character{A,B,...}
matches any of the sub-patterns A, B, etc.\X
matches the character X exactly (even if it’s a special character such as*
)[a-zA-Z0-9\-_]
matches any of the characters froma
toz
, or fromA
toZ
, or from0
to9
, or a hyphen (note the use of\-
to mean an actual hyphen rather than indicating a range of characters), or_
.[!a-z]
matches any character OTHER THAN those froma
toz
.
In addition, it’s possible to cancel an earlier ignore pattern by
putting it in a .dotscience-ignore
file, but prefixed with a -
symbol. For instance, to enable uploading files whose names start with
a .
, simply put -.*
Specifying environment variables: -e NAME
and -e NAME=VALUE
.
You can specify shell environment variables to be in effect when running the command.
-e NAME=VALUE
sets the environment variable NAME to the given VALUE.-e NAME
sets the environment variable NAME to whatever value the environment variable NAME has on the computer where you’re runningds
, in effect passing it through.
Git integration: -R REPO
and -b REF
.
If the code or data you wish to work with originates in a Git repo, you can ask Dotscience to fetch it into your workspace before running the job. Pass the git URL (any URL you could pass to git clone
) in as the REPO and it will be checked out into a subdirectory of the workspace, either named after the repo or called code
if the system can’t deduce that from the repo URL.
By default, it will check out master; specifying a REF (which can be a branch name, a tag name, or an arbitrary commit hash) will cause it to check out that ref instead.
The first time you use this in a workspace, it will git clone
the repo into your workspace. It’s actually saved into the workspace, so subsequent runs will find the checkout there; but it won’t be updated with subsequent changes in the git repo unless you specify -R
and -b
if appropriate.
If you have an SSH key associated with this project, then that ssh key will be available for git to use to access your repo using SSH ([email protected]:PATH
style git URLs). See Managing Secrets below.
Specifying runner labels: -r NAME
and -r NAME=VALUE
.
By default, the job will be run by the first free runner that picks it up. However, by passing runner labels through, you can make sure that only runners satisfying some requirement will try to run your job, or provide configuration for the runner that picks it up. The meanings of the NAMEs and VALUEs depends on the runners your Dotscience account is configured with - consult your administrator for details.
-r NAME=VALUE
sets the runner label NAME to the given VALUE.-r NAME
sets the runner lab NAME to whatever value the environment variable NAME has on the computer where you’re runningds
.
Asynchronous mode: -d
.
If you pass the -d
flag, the ds run
will output a task ID once
execution has been scheduled, then return you to your prompt rather
than waiting for the job to actually run.
This task ID can then be used with the commands listed in the next section, Task Management Commands.
Task Management Commands.
Jobs started with ds run
- be they asynchronous (-d
) or not - as well as JupyterLab instances started through the Dotscience web interface all count as “tasks”, and can be managed with ds
commands.
Listing tasks: ds task ls
.
The ds task ls
command lists all the tasks known to your account. The columns in the output are ID
(the task ID), STATUS
(whether it’s running or in some other state), WORKLOAD
(jupyter
for JupyterLab or command
for ds run
), and AGE
(when the task was created).
Note that terminated tasks remain in the list for several days before being removed, unless manually removed with ds task rm
.
Examining task details: ds task inspect ID
.
ds task inspect ID
returns all the details of a task, as JSON.
Terminating a task: ds task stop ID
.
ds task stop ID
requests that a running task be stopped. It might take a while for the task to finish cleanly - any pending changes need to be uploaded back to the Hub, so watch the task’s status to find out when it’s completed!
Deleting a task: ds task rm ID
.
ds task rm ID
removes a terminated task from the list. It’s only really useful if you want to clean up the list, when you’re sure you won’t want any further information about the task.
Getting task logs: ds logs [-v] [-f] ID
.
ds logs ID
prints out all the stored logs for the task.
If -f
is given, then once it has printed out all stored logs, if the task is still running, it will continue to print out logs as they happen, until the task terminates.
If -v
is given, then all log entries will be printed; otherwise, only log output from your workload will be printed. -v
mode is mainly useful for diagnosing infrastructure issues. In -v
mode, every log entry is prefixed with a header consisting of the timestamp, the log entry type (omitted for workload standard output), a colon, then the line; without -v
, workload standard output lines are emitted as-is, while workload standard error lines have ERROR:
prepended to them.
Managing projects.
Getting a list of projects: ds project ls [-q]
.
ds project ls
lists all the projects in your account, including their ID and name. It’s quite possible to have multiple projects with the same name - if a project is in your account because it was shared with you and then you made a fork of it, for instance, you will have exactly that situation! Therefore, the IDs are often necessary to precisely specify which project you are referring to in commands that take a project name or ID.
This view will also show the datasets which are associated with this project and on what path they are mounted.
If you pass the -q
flag, then rather than a table listing the project ID, NAME, workspace DOTS, COLLABORATORS and AGE columns, just the project names are listed (with no table heading).
Creating a project: ds project create
.
This command creates a new project, and prints its ID out the console.
Describe a project: ds project inspect PROJECT
.
ds project inspect
prints out the full details of a project, in JSON format.
Deleting a project: ds project rm PROJECT
.
This command deletes a project. Don’t run it by accident, as there’s no undelete!
List files in a project: ds file ls [-q] PROJECT
.
To list the files in a project’s workspace, use ds file ls PROJECT
. The result is a table with the file’s NAME, SIZE, and LAST MODIFIED timestamp; but if you specify -q
it just lists the names.
Link a dataset to a project: ds project link PROJECT DATASET PATH
This command links a dataset to a project on PATH, so that when tasks are launched, the dataset is mounted at ./<PATH>
. For example if you were to use the path s3
the path to accessing the contents of the dataset from the task would begin ./s3/
.
Unlink a dataset from a project: ds project unlink PROJECT DATASET
This command unlinks a dataset from a project so that it no longer shows up as a mount when tasks are launched.
Managing datasets
Getting a list of datasets: ds dataset ls [-q]
.
ds dataset ls
lists all the projects in your account, including their ID and name. It will also show the sync status of the dataset, meaning - if it is an aws s3 dataset - how much of the s3 dataset has been downloaded to Dotscience.
If you pass the -q
flag, then rather than a table listing the dataset ID, NAME, etc columns, just the dataset names are listed (with no table heading).
Creating a dataset: ds dataset create s3 --name NAME --access-key-id AWS_ACCESS_KEY --secret-access-key AWS_SECRET_ACCESS_KEY --bucket BUCKET_NAME
.
This command creates a new s3 dataset and prints the ID to the console.
Describe a dataset: ds dataset inspect DATASET
.
ds dataset inspect
prints out the full details of a dataset, in JSON format.
Deleting a dataset: ds dataset rm DATASET
.
This command deletes a dataset. Don’t run it by accident, as there’s no undelete!
Managing runs.
Run listings: ds runs ls --summary-type=SUMMARY PROJECT
.
You can get a list of runs that have happened in a project with ds runs ls PROJECT
(the project name or ID both work, and you’ll need to use the ID if you have multiple projects with the same name). Only runs with a summary statistic of type SUMMARY are listed, and the value of that statistic is shown for every run in the SCORE column; if you omit --summary-type
the system will pick one arbitrarily, which is just what you want if you only have one summary statistic in your project.
Details of a run: ds runs inspect --project PROJECT RUN
.
You can view the full details of a run in a given project with ds runs inspect
. The output is JSON, in the Run Metadata Format.
Managing runners.
List runners: ds runner ls [-q]
.
This command gets a list of all the runners attached to your account. The output is a table listing the ID, NAME, how many RUNNING TASKS it has, its STATUS, its TYPE (CPU or GPU), and its AGE; but if you specify -q
then just the names are listed, with no table heading.
Details of a runner: ds runner inspect RUNNER
.
ds runner inspect RUNNER
returns the full details of the runner in JSON format, including a list of tasks the runner has run or is still running.
Create a runner: ds runner create [-q] [-d DESCRIPTION] [-s SIZE] [-t cpu|gpu-nvidia-runtime] [NAME]
.
This command creates a new runner.
All the fields are optional: if no NAME is given, one will be picked. If no -t
type is specified, cpu
will be assumed. If no SIZE is given for the runner’s storage size in gigabytes, 10 will be assumed.
The output will look something like this:
Runner ID: c871506a-93bb-49df-9ba7-6dca75a476bf
Runner name: 791d4f4a-untitled
Runner type: cpu
Runner API token: PLBMTXVP3RLLTUFOEN3FOBCYDSPHL7BSLDZXVGXX3AAGZTZBBLJQ====
To start a runner, run:
docker pull quay.io/dotmesh/dotscience-runner:0.5.0 && \
docker run --name dotscience-runner -d -e TOKEN=PLBMTXVP3RLLTUFOEN3FOBCYDSPHL7BSLDZXVGXX3AAGZTZBBLJQ==== \
--restart always -v /var/run/docker.sock:/var/run/docker.sock \
-v dotscience-task-spool:/spool \
quay.io/dotmesh/dotscience-runner:0.5.0
The latter part is the command to run that will bring the runner up on any Docker installation. If you use the -q
flag, then that’s all that is output; the metadata and instructional text is suppressed.
Remove a runner: ds runner rm RUNNER
.
This command gets rid of a runner.
Managing secrets.
List secrets: ds secret ls [-q]
.
This will list all the secrets known to your account. It outputs a table listing the ID, NAME, TYPE, PROJECT and AGE; unless you specify -q
, in which case you just get the names.
List details of a secret: ds secret inspect SECRET
.
This prints out the details of a secret, in JSON form. The data
field, for an SSH key, will be base64-encoded JSON containing public_key
and private_key
fields; they can be extracted using jq
like so to get values that can be used with a normal ssh
client:
$ ds secret inspect SECRET | jq -r .[0].data | base64 -d | jq -r .public_key | base64 -d
$ ds secret inspect SECRET | jq -r .[0].data | base64 -d | jq -r .private_key | base64 -d
Generate a secret: ds secret generate --project PROJECT --name NAME
.
This will generate an SSH keypair associated with PROJECT, called NAME, and will output the public part of the keypair.
Removing a secret: ds secret rm SECRET
.
You can delete a secret from your account with ds secret rm SECRET
.
Other Useful Commands.
These aren’t commands you’ll need very often, but here they are for completeness!
Online help: ds help
.
You can obtain online help on a command by typing ds help COMMAND
or
ds COMMAND --help
:
$ ds help version
Display the version of the Dotscience command-line tool
Usage:
ds version [flags]
Flags:
-h, --help help for version
Checking the version of the ds
tool: ds version
.
$ ds version
Dotscience command-line tool version: 0.1
Connecting to different Dotscience hubs: ds set server-url
.
By default, ds
connects to the Dotscience SaaS server, but if you have a dedicated instance of your own, use ds set server-url
to tell ds
to connect to it. Set it to the scheme and domain part of the base URL you use for logging into Dotscience, such as https://cloud.dotscience.com
; do not include anything after the domain name!
Complete reference
This reference is up to date for the Dotscience command-line tool as of April 6th 2020
ds
Dotscience is a platform for managing, running,
and tracking data science workloads.
See https://www.dotscience.com/ for more details.
The ds tool provides access to Dotscience functionality
from your command line.
Usage:
ds [command]
Available Commands:
addon Manage and view addons
ci Manage ci configurations - create, list and modify ci configurations
dataset Manage datasets - create, list and modify datasets
deployer Manage deployers - create, list and modify deployers
deployment Manage deployment - create, list and modify deployments
file View files
help Help about any command
login Login and save credentials
logs Task logs
model Manage models - list and build models
project Manage projects - create, list and modify projects
rebuild-cache Rebuild the cache for the specified project, or if not specified, all projects on the Hub
rp Manage runner profiles - create, list and modify runners (requires cluster admin permissions)
run Run a Dotscience task
runner Manage runners - create, list and modify runners
runs View project runs
secret Manage secrets - create and list SSH keys
set Update configuration settings
task Manage tasks - control tasks
version Display the version of the Dotscience command-line tool
Flags:
-h, --help help for ds
Use "ds [command] --help" for more information about a command.
ds addon
Manage and view addons
Usage:
ds addon [flags]
ds addon [command]
Available Commands:
disable Disable addon for a task - meltano, tensorboard
enable Enable addon for a task - meltano, tensorboard
Flags:
-h, --help help for addon
Use "ds addon [command] --help" for more information about a command.
ds addon disable
Disable addon for a task - meltano, tensorboard
Usage:
ds addon disable [flags]
ds addon disable [command]
Available Commands:
meltano Disable Meltano addon for the running task
tensorboard Disable Tensorboard addon for the running task
Flags:
-h, --help help for disable
Use "ds addon disable [command] --help" for more information about a command.
ds addon disable meltano
Disable Meltano addon for the running task
Usage:
ds addon disable meltano [OPTIONS] [flags]
Flags:
-h, --help help for meltano
ds addon disable tensorboard
Disable Tensorboard addon for the running task
Usage:
ds addon disable tensorboard [OPTIONS] [flags]
Flags:
-h, --help help for tensorboard
ds addon enable
Enable addon for a task - meltano, tensorboard
Usage:
ds addon enable [flags]
ds addon enable [command]
Available Commands:
meltano Enable Meltano addon for the running task
tensorboard Enable Tensorboard addon for the running task
Flags:
-h, --help help for enable
Use "ds addon enable [command] --help" for more information about a command.
ds addon enable meltano
Enable Meltano addon for the running task
Usage:
ds addon enable meltano [OPTIONS] [flags]
Flags:
-h, --help help for meltano
ds addon enable tensorboard
Enable Tensorboard addon for the running task
Usage:
ds addon enable tensorboard [OPTIONS] [flags]
Flags:
-h, --help help for tensorboard
ds ci
Manage ci configurations - create, list and modify ci configurations
Usage:
ds ci [flags]
ds ci [command]
Available Commands:
create Create ci configurations - gitlab
inspect Display detailed information on one or more ci configurations
ls List ci configurations
rm Remove one or more ci configurations
Flags:
-h, --help help for ci
Use "ds ci [command] --help" for more information about a command.
ds ci create
Create ci configurations - gitlab
Usage:
ds ci create [flags]
ds ci create [command]
Available Commands:
gitlab Create a new Gitlab CI configuration
Flags:
-h, --help help for create
Use "ds ci create [command] --help" for more information about a command.
ds ci create gitlab
Create a new Gitlab CI configuration
Usage:
ds ci create gitlab [OPTIONS] [flags]
Flags:
--default-builder Set this configuration as the default builder for your account. Default false
--description string Description of config
-h, --help help for gitlab
--project string name of the gitlab project
--project-id string ID of the gitlab project
--ref string git ref to run this on, e.g master
--token string gitlab token to use for authentication
--trigger-token string token used to trigger the pipeline
--url string gitlab instance URL
ds ci inspect
Display detailed information on one or more ci configurations
Usage:
ds ci inspect [OPTIONS] CONFIGURATION [CONFIGURATION...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds ci ls
List ci configurations
Usage:
ds ci ls [OPTIONS] [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds ci rm
Remove one or more ci configuration.
Usage:
ds ci rm [OPTIONS] CI_CONFIG [CI_CONFIG...] [flags]
Aliases:
rm, remove
Examples:
$ ds ci rm prodConfig
prodConfig
Flags:
-h, --help help for rm
ds dataset
Manage datasets - create, list and modify datasets
Usage:
ds dataset [flags]
ds dataset [command]
Available Commands:
create Create datasets - s3
inspect Display detailed information on one or more datasets
ls List datasets
rm Remove one or more datasets
Flags:
-h, --help help for dataset
Use "ds dataset [command] --help" for more information about a command.
ds dataset create
Create datasets - s3
Usage:
ds dataset create [flags]
ds dataset create [command]
Available Commands:
s3 Create a new s3 dataset
Flags:
-h, --help help for create
Use "ds dataset create [command] --help" for more information about a command.
ds dataset create s3
Create a new s3 dataset
Usage:
ds dataset create s3 [OPTIONS] [flags]
Flags:
--access-key-id string AWS programmatic access key ID - account must have access to the bucket
--bucket string Name of s3 bucket
-h, --help help for s3
--secret-access-key string AWS programmatic secret access key - must be a matching pair with access key ID and have access to the bucket
ds dataset inspect
Display detailed information on one or more datasets
Usage:
ds dataset inspect [OPTIONS] DATASET [DATASET...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds dataset ls
List datasets
Usage:
ds dataset ls [OPTIONS] [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds dataset rm
Remove one or more dataset.
Usage:
ds dataset rm [OPTIONS] DATASET [DATASET...] [flags]
Aliases:
rm, remove
Examples:
$ ds dataset rm prodDataset
prodDataset
Flags:
-h, --help help for rm
ds deployer
Manage deployers - create, list and modify deployers
Usage:
ds deployer [flags]
ds deployer [command]
Available Commands:
create Create a new deployer
inspect Display detailed information on one or more deployers
ls List deployers
rm Remove one or more deployers
Flags:
-h, --help help for deployer
Use "ds deployer [command] --help" for more information about a command.
ds deployer create
Create a new deployer
Usage:
ds deployer create [OPTIONS] [flags]
Flags:
--custom-hostnames Whether to allow custom deployment hostnames for this deployer
-d, --description string Description
-h, --help help for create
--managed Managed deployer (shared with other cluster users)
--max-replicas int Max number of replicas allowed for the deployment (default 2)
--namespace string Deployment namespace (default "default")
-q, --quiet Quiet mode (just output the runner startup command)
--subdomain string Subdomain to use for new deployments (for example: '.app.cloud.dotscience.net')
-t, --token string Deployer token (optional)
ds deployer inspect
Display detailed information on one or more deployers
Usage:
ds deployer inspect [OPTIONS] DEPLOYER [DEPLOYER...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds deployer ls
List deployers
Usage:
ds deployer ls [OPTIONS] [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds deployer rm
Remove one or more deployers.
Usage:
ds deployer rm [OPTIONS] DEPLOYER [DEPLOYER...] [flags]
Aliases:
rm, remove
Examples:
$ ds deployer rm prodDeployer
prodDeployer
Flags:
-h, --help help for rm
ds deployment
Manage deployment - create, list and modify deployments
Usage:
ds deployment [flags]
ds deployment [command]
Available Commands:
create Create a new deployment
inspect Display detailed information on one or more deployments
ls List deployments
rm Remove one or more deployments
Flags:
-h, --help help for deployment
Use "ds deployment [command] --help" for more information about a command.
ds deployment create
Create a new deployment
Usage:
ds deployment create [OPTIONS] [flags]
Flags:
-d, --deployer string Deployer to use
-h, --help help for create
--host string Hostname for ingress controller
-i, --image-name string Docker image name
--ingress-class string Ingress class to use (default "webrelay")
--model-classes string Path to model classes file
--model-name string Model name
--namespace string Namespace
-p, --port int Docker container port (default 8501)
-q, --quiet Quiet mode (just output the runner startup command)
-r, --replicas int Replicas to run (number of containers to start) (default 1)
ds deployment inspect
Display detailed information on one or more deployments
Usage:
ds deployment inspect [OPTIONS] DEPLOYMENT [DEPLOYMENT...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds deployment ls
List deployments
Usage:
ds deployment ls [OPTIONS] [flags]
Aliases:
ls, list
Flags:
-d, --deployer string Deployer ID or name
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds deployment logs
Error: "ds deployment logs" requires 1 argument(s).
See 'ds deployment logs --help'.
Usage: ds deployment logs [OPTIONS] DEPLOYMENT [flags]
Display deployment logs
Usage:
ds deployment logs [OPTIONS] DEPLOYMENT [flags]
Example:
$ ds deployment logs 3712682c-a201-491a-84db-595871d3d86e
2020-04-06 12:36:55.790737: I tensorflow_serving/model_servers/server.cc:86] Building single TensorFlow model file config: model_name: model model_base_path: /models/model
2020-04-06 12:36:55.791019: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2020-04-06 12:36:55.791039: I tensorflow_serving/model_servers/server_core.cc:573] (Re-)adding model: model
2020-04-06 12:36:55.891407: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: model version: 1}
2020-04-06 12:36:56.035047: I tensorflow_serving/model_servers/server.cc:378] Exporting HTTP/REST API at:localhost:8501 ...
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
ds deployment rm
Remove one or more deployments.
Usage:
ds deployment rm [OPTIONS] DEPLOYMENT [DEPLOYMENT...] [flags]
Aliases:
rm, remove
Examples:
$ ds deployment rm model-staging
odel-staging
Flags:
-h, --help help for rm
ds file
View files
Usage:
ds file [flags]
ds file [command]
Available Commands:
download Download files from a project
ls List files for a project
upload Upload file to a project
Flags:
-h, --help help for file
Use "ds file [command] --help" for more information about a command.
ds file download
Download files from a project
Usage:
ds file download [OPTIONS] PATH DESTINATION [flags]
Aliases:
download, download
Flags:
-c, --commit string Commit ID (default "latest")
-d, --destination string Destination where to download the file or directory
-f, --force Overwrite destination
-h, --help help for download
--path string File or directory to download
-p, --project string Project ID or name
ds file ls
List files for a project
Usage:
ds file ls [OPTIONS] PROJECT [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-p, --project string Project ID or name
-q, --quiet Only display names
ds file upload
Upload file to a project
Usage:
ds file upload [OPTIONS] FILEPATH [flags]
Aliases:
upload, upload
Flags:
-f, --filename string Filename
-h, --help help for upload
--path string Override filename (for example if you want to upload file to a specific directory)
-p, --project string Project ID or name
ds help
Help provides help for any command in the application.
Simply type ds help [path to command] for full details.
Usage:
ds help [command] [flags]
Flags:
-h, --help help for help
ds login
Login and save credentials
Usage:
ds login [OPTIONS] [flags]
Flags:
-h, --help help for login
-p, --password string Password
-u, --username string Username.
ds logs
Task logs
Usage:
ds logs [OPTIONS] TASK [flags]
Flags:
-f, --follow Follow logs output
--format string Pretty-print using a Go template
-h, --help help for logs
-q, --quiet Only display names
--task string Task ID
-v, --verbose Display verbose logs
ds model
Manage models - list and build models
Usage:
ds model [flags]
ds model [command]
Available Commands:
build Build model image
builds Build list for a model
inspect Display detailed information on one or more models
ls List models
Flags:
-h, --help help for model
Use "ds model [command] --help" for more information about a command.
## ds model ls
Lists all available models for a logged in user.
Example
ds model ls
ID NAME FRAMEWORK BUILD STATUS AGE
20fe31f5-7df6-4fc4-9ad1-1d70bc146d1a roadsigns tensorflow-model completed (quay.io/dotscience-playground/models:77da5825-a4c4-41b2-95eb-34280c4c688a-12478713-27c4-4bbe-985d-8ce69a2199c5) 5 days
d71e35b8-9cfd-492b-a708-8a825ab5e639 mnist-epochs-1 tensorflow-model completed (quay.io/dotscience-playground/models:77da5825-a4c4-41b2-95eb-34280c4c688a-acf569a3-e355-48ad-8622-901d2f5629da) 13 days
ds model build
Builds a model into a docker image. Supported model frameworks are tensorflow and scikit-learn.
Usage: ds model build [OPTIONS] MODEL [flags]
Build model image
Usage:
ds model build [OPTIONS] MODEL [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for build
ds model builds
List of builds for a given model.
Usage: ds model builds [OPTIONS] MODEL [flags]
Build list for a model
Usage:
ds model builds [OPTIONS] MODEL [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for builds
-q, --quiet Only display IDs
ds project
Manage projects - create, list and modify projects
Usage:
ds project [flags]
ds project [command]
Available Commands:
create Create a new project
inspect Display detailed information on one or more projects
link link a project to a dataset on a given path
link unlink a dataset from a project
ls List projects
rm Remove one or more projects
Flags:
-h, --help help for project
Use "ds project [command] --help" for more information about a command.
ds project create
Create a new project
Usage:
ds project create [OPTIONS] [flags]
Flags:
-h, --help help for create
ds project inspect
Display detailed information on one or more projects
Usage:
ds project inspect [OPTIONS] PROJECT [PROJECT...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds project link
link a project to a dataset on a given path
Usage:
ds project link ID DATASET PATH [flags]
Flags:
-h, --help help for link
ds project ls
List projects
Usage:
ds project ls [OPTIONS] [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds project rm
Remove one or more projects.
Usage:
ds project rm [OPTIONS] PROJECT [PROJECT...] [flags]
Aliases:
rm, remove
Examples:
$ ds project rm my-project
my-project
Flags:
-h, --help help for rm
ds rp
Manage runner profiles - create, list and modify runners (requires cluster admin permissions)
Usage:
ds rp [flags]
ds rp [command]
Available Commands:
create Create a new runner profile
inspect Display detailed information on one or more runner profiles
ls List runner profiles
rm Remove one or more runner profiles
update Update an existing runner profile
Flags:
-h, --help help for rp
Use "ds rp [command] --help" for more information about a command.
ds rp create
Create a new runner profile
Usage:
ds rp create [OPTIONS] [flags]
Flags:
-d, --description string Description
--disk-size int Disk size (default 200)
--gpu-count int GPU count
--gpu-type string GPU type (if any), you can also choose machine size that includes GPU
-h, --help help for create
--machine-image string Machine image (AMI) - optional
--machine-image-family string Machine image family (AMI), used by GCP
--preemtible Preemtible (if supported by the platform)
--pricing string Optional pricing info for the users to see
-r, --runtime string Runner runtime type (cpu, gpu-nvidia-runtime or gpu-nvidia-docker) (default "cpu")
--size string Machine size (for example 'p3.2xlarge' on AWS)
ds rp inspect
Display detailed information on one or more runner profiles
Usage:
ds rp inspect [OPTIONS] PROFILE [PROFILE...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds rp ls
List runner profiles
Usage:
ds rp ls [OPTIONS] [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds rp rm
Remove one or more runner profiles.
Usage:
ds rp rm [OPTIONS] PROFILE [PROFILE...] [flags]
Aliases:
rm, remove
Examples:
$ ds rp rm gpu-profile
gpu-profile
Flags:
-h, --help help for rm
ds rp update
Update an existing runner profile
Usage:
ds rp update [OPTIONS] [flags]
Flags:
-d, --description string Description
--disk-size int Disk size (default 200)
--gpu-count int GPU count
--gpu-type string GPU type (if any), you can also choose machine size that includes GPU
-h, --help help for update
--machine-image string Machine image (AMI) - optional
--machine-image-family string Machine image family (AMI), used by GCP
--preemtible Preemtible (if supported by the platform)
--pricing string Optional pricing info for the users to see
--size string Machine size (for example 'p3.2xlarge' on AWS)
ds run
Run a Dotscience task
Usage:
ds run [OPTIONS] COMMAND [COMMAND...] [flags]
Flags:
-d, --async Run the command in the background.
-c, --command stringArray Command to run
-e, --environment NAME=VALUE Set a variable of the form NAME=VALUE in the shell environment when running the command
-h, --help help for run
-I, --image string Docker image to use. If not set, will use the latest stable official dotscience python image
--nvidia Use the nvidia GPU runtime
-p, --project-name string Project name to use
-b, --ref git checkout Branch, tag, or commit reference to start from in git (i.e a valid entry to git checkout). If this is not set and repo is set, will checkout master and pull the latest commit.
-R, --repo string Git repo to link to. If not set, will not work with git.
-r, --runner KEY=VALUE Specify KEY=VALUE labels to configure the runner.
-S, --selector string Selector to pick a specific runner, eg 'id=RUNNER ID'
-u, --upload-path string Upload the contents of the supplied directory while doing the run. If not supplied, will not do any uploads.
-v, --verbose Display all internal system logging
ds runner
Manage runners - create, list and modify runners
Usage:
ds runner [flags]
ds runner [command]
Available Commands:
create Create a new runner
inspect Display detailed information on one or more runners
ls List runners
rm Remove one or more runner
show-startup-script Display the startup script for a runner
Flags:
-h, --help help for runner
Use "ds runner [command] --help" for more information about a command.
ds runner create
Create a new runner
Usage:
ds runner create [OPTIONS] [flags]
Flags:
-d, --description string Description
-h, --help help for create
-s, --pool-size int Pool size (GB) (default 10)
-q, --quiet Quiet mode (just output the runner startup command)
-t, --runner-runtime string Runner runtime type, defaults to 'cpu' (cpu or gpu-nvidia-runtime for GPU runners) (default "cpu")
ds runner inspect
Display detailed information on one or more runners
Usage:
ds runner inspect [OPTIONS] RUNNER [RUNNER...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds runner ls
List runners
Usage:
ds runner ls [OPTIONS] [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds runner rm
Remove one or more runners.
Usage:
ds runner rm [OPTIONS] RUNNER [RUNNER...] [flags]
Aliases:
rm, remove
Examples:
$ ds runner rm my-runner
my-runner
Flags:
-h, --help help for rm
ds runner show-startup-script
Display the startup script for a runner
Usage:
ds runner show-startup-script [OPTIONS] RUNNER [flags]
Flags:
-h, --help help for show-startup-script
ds runs
View project runs
Usage:
ds runs [flags]
ds runs [command]
Available Commands:
inspect Display detailed information on one or more project runs
ls List runs for a project
Flags:
-h, --help help for runs
Use "ds runs [command] --help" for more information about a command.
ds runs inspect
Display detailed information on one or more project runs
Usage:
ds runs inspect [OPTIONS] RUN [RUN...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
--project string Project ID or name
ds runs ls
List runs for a project
Usage:
ds runs ls [OPTIONS] PROJECT [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
--project string Project ID or name
-q, --quiet Only display names
--summary-type string Summary type
-v, --verbose Verbose - display more information
ds secret
Manage secrets - create and list SSH keys
Usage:
ds secret [flags]
ds secret [command]
Available Commands:
generate Generate a public and private key SSH pair
inspect Display detailed information on one or more secrets
ls List secrets
rm Remove one or more secrets
Flags:
-h, --help help for secret
Use "ds secret [command] --help" for more information about a command.
ds secret generate
Generate a public and private key SSH pair
Usage:
ds secret generate [OPTIONS] [flags]
Flags:
-h, --help help for generate
--name string Secret name
-p, --project string Project ID
ds secret inspect
Display detailed information on one or more secrets
Usage:
ds secret inspect [OPTIONS] SECRET [SECRET...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds secret ls
List secrets
Usage:
ds secret ls [OPTIONS] [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds secret rm
Remove one or more secrets.
Usage:
ds secret rm [OPTIONS] SECRET [SECRET...] [flags]
Aliases:
rm, remove
Examples:
$ ds secret rm secret-name-or-id
secret-name-or-id
Flags:
-h, --help help for rm
ds set
Update configuration settings
Usage:
ds set [command]
Available Commands:
agent-image Update the Dotscience Agent docker image
default-namespace Update the default namespace
server-url Update the Dotscience Server URL
Flags:
-h, --help help for set
Use "ds set [command] --help" for more information about a command.
ds set agent-image
Update the Dotscience Agent docker image
Usage:
ds set agent-image <docker image> [flags]
Flags:
-h, --help help for agent-image
ds set default-namespace
Update the default namespace
Usage:
ds set default-namespace <namespace> [flags]
Flags:
-h, --help help for default-namespace
ds set server-url
Update the Dotscience Server URL
Usage:
ds set server-url <url> [flags]
Flags:
-h, --help help for server-url
ds task
Manage tasks - control tasks
Usage:
ds task [flags]
ds task [command]
Available Commands:
inspect Display detailed information on one or more tasks
ls List tasks for the runner
rm Remove one or more tasks
stop Stop (terminate) one or more tasks
Flags:
-h, --help help for task
Use "ds task [command] --help" for more information about a command.
ds task inspect
Display detailed information on one or more tasks
Usage:
ds task inspect [OPTIONS] TASK [TASK...] [flags]
Flags:
-f, --format string Format the output using the given Go template
-h, --help help for inspect
ds task ls
List tasks for the runner
Usage:
ds task ls [OPTIONS] RUNNER [RUNNER...] [flags]
Aliases:
ls, list
Flags:
--format string Pretty-print using a Go template
-h, --help help for ls
-q, --quiet Only display names
ds task rm
Remove one or more tasks.
Usage:
ds task rm [OPTIONS] TASK [TASK...] [flags]
Aliases:
rm, remove
Examples:
$ ds task rm my-task-uuid
my-task-uuid
Flags:
-h, --help help for rm
ds task stop
Stop one or more tasks. Tasks cannot be restarted.
Usage:
ds task stop [OPTIONS] TASK [TASK...] [flags]
Aliases:
stop, remove
Examples:
$ ds task stop task-uuid
task-uuid
Flags:
-h, --help help for stop
ds version
Display the version of the Dotscience command-line tool
Usage:
ds version [flags]
Flags:
-h, --help help for version