Command line reference

Reference for the Dotscience command line interface

All commands begin with ds, then contain a command, which in turn may contain further subcommands, e.g.

ds
ds deployer
ds deployer create

Help for each command can be found by appending -h, --help, or referring to the below.

This reference is up to date for the Dotscience command-line tool as of Feb 10th 2020, git commit fb066e7.

ds

Dotscience is a platform for managing, running,
and tracking data science workloads.

See https://www.dotscience.com/ for more details.

The ds tool provides access to Dotscience functionality
from your command line.

Usage:
  ds [command]

Available Commands:
  addon       Manage and view addons
  ci          Manage ci configurations - create, list and modify ci configurations
  dataset     Manage datasets - create, list and modify datasets
  deployer    Manage deployers - create, list and modify deployers
  deployment  Manage deployment - create, list and modify deployments
  file        View files
  help        Help about any command
  login       Login and save credentials
  logs        Task logs
  project     Manage projects - create, list and modify projects
  rp          Manage runner profiles - create, list and modify runners (requires cluster admin permissions)
  run         Run a Dotscience task
  runner      Manage runners - create, list and modify runners
  runs        View project runs
  secret      Manage secrets - create and list SSH keys
  set         Update configuration settings
  task        Manage tasks - control tasks
  version     Display the version of the Dotscience command-line tool

Flags:
  -h, --help   help for ds

Use "ds [command] --help" for more information about a command.

ds addon

Manage and view addons

Usage:
  ds addon [flags]
  ds addon [command]

Available Commands:
  disable     Disable addon for a task - meltano, tensorboard
  enable      Enable addon for a task - meltano, tensorboard

Flags:
  -h, --help   help for addon

Use "ds addon [command] --help" for more information about a command.

ds addon disable

Disable addon for a task - meltano, tensorboard

Usage:
  ds addon disable [flags]
  ds addon disable [command]

Available Commands:
  meltano     Disable Meltano addon for the running task
  tensorboard Disable Tensorboard addon for the running task

Flags:
  -h, --help   help for disable

Use "ds addon disable [command] --help" for more information about a command.

ds addon enable

Enable addon for a task - meltano, tensorboard

Usage:
  ds addon enable [flags]
  ds addon enable [command]

Available Commands:
  meltano     Enable Meltano addon for the running task
  tensorboard Enable Tensorboard addon for the running task

Flags:
  -h, --help   help for enable

Use "ds addon enable [command] --help" for more information about a command.

ds ci

Manage ci configurations - create, list and modify ci configurations

Usage:
  ds ci [flags]
  ds ci [command]

Available Commands:
  create      Create ci configurations - gitlab
  inspect     Display detailed information on one or more ci configurations
  ls          List ci configurations
  rm          Remove one or more ci configurations

Flags:
  -h, --help   help for ci

Use "ds ci [command] --help" for more information about a command.

ds ci create

Create ci configurations - gitlab

Usage:
  ds ci create [flags]
  ds ci create [command]

Available Commands:
  gitlab      Create a new Gitlab CI configuration

Flags:
  -h, --help   help for create

Use "ds ci create [command] --help" for more information about a command.

ds ci create gitlab

Create a new Gitlab CI configuration

Usage:
  ds ci create gitlab [OPTIONS] [flags]

Flags:
      --default-builder        Set this configuration as the default builder for your account. Default false
      --description string     Description of config
  -h, --help                   help for gitlab
      --project string         name of the gitlab project
      --project-id string      ID of the gitlab project
      --ref string             git ref to run this on, e.g master
      --token string           gitlab token to use for authentication
      --trigger-token string   token used to trigger the pipeline
      --url string             gitlab instance URL

ds ci inspect

Display detailed information on one or more ci configurations

Usage:
  ds ci inspect [OPTIONS] CONFIGURATION [CONFIGURATION...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds ci ls

List ci configurations

Usage:
  ds ci ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds ci rm

Remove one or more ci configuration.

Usage:
  ds ci rm [OPTIONS] CI_CONFIG [CI_CONFIG...] [flags]

Aliases:
  rm, remove

Examples:

$ ds ci rm prodConfig
prodConfig

Flags:
  -h, --help   help for rm

ds dataset

Manage datasets - create, list and modify datasets

Usage:
  ds dataset [flags]
  ds dataset [command]

Available Commands:
  create      Create datasets - s3
  inspect     Display detailed information on one or more datasets
  ls          List datasets
  rm          Remove one or more datasets

Flags:
  -h, --help   help for dataset

Use "ds dataset [command] --help" for more information about a command.

ds dataset create

Create datasets - s3

Usage:
  ds dataset create [flags]
  ds dataset create [command]

Available Commands:
  s3          Create a new s3 dataset

Flags:
  -h, --help   help for create

Use "ds dataset create [command] --help" for more information about a command.

ds dataset create s3

Create a new s3 dataset

Usage:
  ds dataset create s3 [OPTIONS] [flags]

Flags:
      --access-key-id string       AWS programmatic access key ID - account must have access to the bucket
      --bucket string              Name of s3 bucket
  -h, --help                       help for s3
      --secret-access-key string   AWS programmatic secret access key - must be a matching pair with access key ID and have access to the bucket

ds dataset inspect

Display detailed information on one or more datasets

Usage:
  ds dataset inspect [OPTIONS] DATASET [DATASET...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds dataset ls

List datasets

Usage:
  ds dataset ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds dataset rm

Remove one or more dataset.

Usage:
  ds dataset rm [OPTIONS] DATASET [DATASET...] [flags]

Aliases:
  rm, remove

Examples:

$ ds dataset rm prodDataset
prodDataset

Flags:
  -h, --help   help for rm

ds deployer

Manage deployers - create, list and modify deployers

Usage:
  ds deployer [flags]
  ds deployer [command]

Available Commands:
  create      Create a new deployer
  inspect     Display detailed information on one or more deployers
  ls          List deployers
  rm          Remove one or more deployers

Flags:
  -h, --help   help for deployer

Use "ds deployer [command] --help" for more information about a command.

ds deployer create

Create a new deployer

Usage:
  ds deployer create [OPTIONS] [flags]

Flags:
      --custom-hostnames     Whether to allow custom deployment hostnames for this deployer
  -d, --description string   Description
  -h, --help                 help for create
      --managed              Managed deployer (shared with other cluster users)
      --max-replicas int     Max number of replicas allowed for the deployment (default 2)
      --namespace string     Deployment namespace (default "default")
  -q, --quiet                Quiet mode (just output the runner startup command)
      --subdomain string     Subdomain to use for new deployments (for example: '.app.cloud.dotscience.net')
  -t, --token string         Deployer token (optional)

ds deployer inspect

Display detailed information on one or more deployers

Usage:
  ds deployer inspect [OPTIONS] DEPLOYER [DEPLOYER...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds deployer ls

List deployers

Usage:
  ds deployer ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds deployer rm

Remove one or more deployers.

Usage:
  ds deployer rm [OPTIONS] DEPLOYER [DEPLOYER...] [flags]

Aliases:
  rm, remove

Examples:

$ ds deployer rm prodDeployer
prodDeployer

Flags:
  -h, --help   help for rm

ds deployment

Manage deployment - create, list and modify deployments

Usage:
  ds deployment [flags]
  ds deployment [command]

Available Commands:
  create      Create a new deployment
  inspect     Display detailed information on one or more deployments
  ls          List deployments
  rm          Remove one or more deployments

Flags:
  -h, --help   help for deployment

Use "ds deployment [command] --help" for more information about a command.

ds deployment create

Create a new deployment

Usage:
  ds deployment create [OPTIONS] [flags]

Flags:
  -d, --deployer string        Deployer to use
  -h, --help                   help for create
      --host string            Hostname for ingress controller
  -i, --image-name string      Docker image name
      --ingress-class string   Ingress class to use (default "webrelay")
      --model-classes string   Path to model classes file
      --model-name string      Model name
      --namespace string       Namespace
  -p, --port int               Docker container port (default 8501)
  -q, --quiet                  Quiet mode (just output the runner startup command)
  -r, --replicas int           Replicas to run (number of containers to start) (default 1)

ds deployment inspect

Display detailed information on one or more deployments

Usage:
  ds deployment inspect [OPTIONS] DEPLOYMENT [DEPLOYMENT...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds deployment ls

List deployments

Usage:
  ds deployment ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
  -d, --deployer string   Deployer ID or name
      --format string     Pretty-print using a Go template
  -h, --help              help for ls
  -q, --quiet             Only display names

ds deployment rm

Remove one or more deployments.

Usage:
  ds deployment rm [OPTIONS] DEPLOYMENT [DEPLOYMENT...] [flags]

Aliases:
  rm, remove

Examples:

$ ds deployment rm model-staging
odel-staging

Flags:
  -h, --help   help for rm

ds file

View files

Usage:
  ds file [flags]
  ds file [command]

Available Commands:
  download    Download files from a project
  ls          List files for a project
  upload      Upload file to a project

Flags:
  -h, --help   help for file

Use "ds file [command] --help" for more information about a command.

ds file download

Download files from a project

Usage:
  ds file download [OPTIONS] PATH DESTINATION [flags]

Aliases:
  download, download

Flags:
  -c, --commit string        Commit ID (default "latest")
  -d, --destination string   Destination where to download the file or directory
  -f, --force                Overwrite destination
  -h, --help                 help for download
      --path string          File or directory to download
  -p, --project string       Project ID or name

ds file ls

List files for a project

Usage:
  ds file ls [OPTIONS] PROJECT [flags]

Aliases:
  ls, list

Flags:
      --format string    Pretty-print using a Go template
  -h, --help             help for ls
  -p, --project string   Project ID or name
  -q, --quiet            Only display names

ds file upload

Upload file to a project

Usage:
  ds file upload [OPTIONS] FILEPATH [flags]

Aliases:
  upload, upload

Flags:
  -f, --filename string   Filename
  -h, --help              help for upload
      --path string       Override filename (for example if you want to upload file to a specific directory)
  -p, --project string    Project ID or name

ds help

Help provides help for any command in the application.
Simply type ds help [path to command] for full details.

Usage:
  ds help [command] [flags]

Flags:
  -h, --help   help for help

ds login

Login and save credentials

Usage:
  ds login [OPTIONS] [flags]

Flags:
  -h, --help              help for login
  -p, --password string   Password
  -u, --username string   Username.

ds logs

Task logs

Usage:
  ds logs [OPTIONS] TASK [flags]

Flags:
  -f, --follow          Follow logs output
      --format string   Pretty-print using a Go template
  -h, --help            help for logs
  -q, --quiet           Only display names
      --task string     Task ID
  -v, --verbose         Display verbose logs

ds project

Manage projects - create, list and modify projects

Usage:
  ds project [flags]
  ds project [command]

Available Commands:
  create      Create a new project
  inspect     Display detailed information on one or more projects
  link        link a project to a dataset on a given path
  link        unlink a dataset from a project
  ls          List projects
  rm          Remove one or more projects

Flags:
  -h, --help   help for project

Use "ds project [command] --help" for more information about a command.

ds project create

Create a new project

Usage:
  ds project create [OPTIONS] [flags]

Flags:
  -h, --help   help for create

ds project inspect

Display detailed information on one or more projects

Usage:
  ds project inspect [OPTIONS] PROJECT [PROJECT...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect
link a project to a dataset on a given path

Usage:
  ds project link ID DATASET PATH [flags]

Flags:
-h, --help   help for link

ds project ls

List projects

Usage:
  ds project ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds project rm

Remove one or more projects.

Usage:
  ds project rm [OPTIONS] PROJECT [PROJECT...] [flags]

Aliases:
  rm, remove

Examples:

$ ds project rm my-project
my-project

Flags:
  -h, --help   help for rm

ds rp

Manage runner profiles - create, list and modify runners (requires cluster admin permissions)

Usage:
  ds rp [flags]
  ds rp [command]

Available Commands:
  create      Create a new runner profile
  inspect     Display detailed information on one or more runner profiles
  ls          List runner profiles
  rm          Remove one or more runner profiles
  update      Update an existing runner profile

Flags:
  -h, --help   help for rp

Use "ds rp [command] --help" for more information about a command.

ds rp create

Create a new runner profile

Usage:
  ds rp create [OPTIONS] [flags]

Flags:
  -d, --description string            Description
      --disk-size int                 Disk size (default 200)
      --gpu-count int                 GPU count
      --gpu-type string               GPU type (if any), you can also choose machine size that includes GPU
  -h, --help                          help for create
      --machine-image string          Machine image (AMI) - optional
      --machine-image-family string   Machine image family (AMI), used by GCP
      --preemtible                    Preemtible (if supported by the platform)
      --pricing string                Optional pricing info for the users to see
  -r, --runtime string                Runner runtime type (cpu, gpu-nvidia-runtime or gpu-nvidia-docker) (default "cpu")
      --size string                   Machine size (for example 'p3.2xlarge' on AWS)

ds rp inspect

Display detailed information on one or more runner profiles

Usage:
  ds rp inspect [OPTIONS] PROFILE [PROFILE...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds rp ls

List runner profiles

Usage:
  ds rp ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds rp rm

Remove one or more runner profiles.

Usage:
  ds rp rm [OPTIONS] PROFILE [PROFILE...] [flags]

Aliases:
  rm, remove

Examples:

$ ds rp rm gpu-profile
gpu-profile

Flags:
  -h, --help   help for rm

ds rp update

Update an existing runner profile

Usage:
  ds rp update [OPTIONS] [flags]

Flags:
  -d, --description string            Description
      --disk-size int                 Disk size (default 200)
      --gpu-count int                 GPU count
      --gpu-type string               GPU type (if any), you can also choose machine size that includes GPU
  -h, --help                          help for update
      --machine-image string          Machine image (AMI) - optional
      --machine-image-family string   Machine image family (AMI), used by GCP
      --preemtible                    Preemtible (if supported by the platform)
      --pricing string                Optional pricing info for the users to see
      --size string                   Machine size (for example 'p3.2xlarge' on AWS)

ds run

Run a Dotscience task

Usage:
  ds run [OPTIONS] COMMAND [COMMAND...] [flags]

Flags:
  -d, --async                    Run the command in the background.
  -c, --command stringArray      Command to run
  -e, --environment NAME=VALUE   Set a variable of the form NAME=VALUE in the shell environment when running the command
  -h, --help                     help for run
  -I, --image string             Docker image to use. If not set, will use the latest stable official dotscience python image
      --nvidia                   Use the nvidia GPU runtime
  -p, --project-name string      Project name to use
  -b, --ref git checkout         Branch, tag, or commit reference to start from in git (i.e a valid entry to git checkout). If this is not set and repo is set, will checkout master and pull the latest commit.
  -R, --repo string              Git repo to link to. If not set, will not work with git.
  -r, --runner KEY=VALUE         Specify KEY=VALUE labels to configure the runner.
  -S, --selector string          Selector to pick a specific runner, eg 'id=RUNNER ID'
  -u, --upload-path string       Upload the contents of the supplied directory while doing the run. If not supplied, will not do any uploads.
  -v, --verbose                  Display all internal system logging

ds runner

Manage runners - create, list and modify runners

Usage:
  ds runner [flags]
  ds runner [command]

Available Commands:
  create              Create a new runner
  inspect             Display detailed information on one or more runners
  ls                  List runners
  rm                  Remove one or more runner
  show-startup-script Display the startup script for a runner

Flags:
  -h, --help   help for runner

Use "ds runner [command] --help" for more information about a command.

ds runner create

Create a new runner

Usage:
  ds runner create [OPTIONS] [flags]

Flags:
  -d, --description string      Description
  -h, --help                    help for create
  -s, --pool-size int           Pool size (GB) (default 10)
  -q, --quiet                   Quiet mode (just output the runner startup command)
  -t, --runner-runtime string   Runner runtime type, defaults to 'cpu' (cpu or gpu-nvidia-runtime for GPU runners) (default "cpu")

ds runner inspect

Display detailed information on one or more runners

Usage:
  ds runner inspect [OPTIONS] RUNNER [RUNNER...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds runner ls

List runners

Usage:
  ds runner ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds runner rm

Remove one or more runners.

Usage:
  ds runner rm [OPTIONS] RUNNER [RUNNER...] [flags]

Aliases:
  rm, remove

Examples:

$ ds runner rm my-runner
my-runner

Flags:
  -h, --help   help for rm

ds runner show-startup-script

Display the startup script for a runner

Usage:
  ds runner show-startup-script [OPTIONS] RUNNER [flags]

Flags:
  -h, --help   help for show-startup-script

ds runs

View project runs

Usage:
  ds runs [flags]
  ds runs [command]

Available Commands:
  inspect     Display detailed information on one or more project runs
  ls          List runs for a project

Flags:
  -h, --help   help for runs

Use "ds runs [command] --help" for more information about a command.

ds runs inspect

Display detailed information on one or more project runs

Usage:
  ds runs inspect [OPTIONS] RUN [RUN...] [flags]

Flags:
  -f, --format string    Format the output using the given Go template
  -h, --help             help for inspect
      --project string   Project ID or name

ds runs ls

List runs for a project

Usage:
  ds runs ls [OPTIONS] PROJECT [flags]

Aliases:
  ls, list

Flags:
      --format string         Pretty-print using a Go template
  -h, --help                  help for ls
      --project string        Project ID or name
  -q, --quiet                 Only display names
      --summary-type string   Summary type
  -v, --verbose               Verbose - display more information

ds secret

Manage secrets - create and list SSH keys

Usage:
  ds secret [flags]
  ds secret [command]

Available Commands:
  generate    Generate a public and private key SSH pair
  inspect     Display detailed information on one or more secrets
  ls          List secrets
  rm          Remove one or more secrets

Flags:
  -h, --help   help for secret

Use "ds secret [command] --help" for more information about a command.

ds secret generate

Generate a public and private key SSH pair

Usage:
  ds secret generate [OPTIONS] [flags]

Flags:
  -h, --help             help for generate
      --name string      Secret name
  -p, --project string   Project ID

ds secret inspect

Display detailed information on one or more secrets

Usage:
  ds secret inspect [OPTIONS] SECRET [SECRET...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds secret ls

List secrets

Usage:
  ds secret ls [OPTIONS] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds secret rm

Remove one or more secrets.

Usage:
  ds secret rm [OPTIONS] SECRET [SECRET...] [flags]

Aliases:
  rm, remove

Examples:

$ ds secret rm secret-name-or-id
secret-name-or-id

Flags:
  -h, --help   help for rm

ds set

Update configuration settings

Usage:
  ds set [command]

Available Commands:
  agent-image       Update the Dotscience Agent docker image
  default-namespace Update the default namespace
  server-url        Update the Dotscience Server URL

Flags:
  -h, --help   help for set

Use "ds set [command] --help" for more information about a command.

ds set agent-image

Update the Dotscience Agent docker image

Usage:
  ds set agent-image <docker image> [flags]

Flags:
  -h, --help   help for agent-image

ds set default-namespace

Update the default namespace

Usage:
  ds set default-namespace <namespace> [flags]

Flags:
  -h, --help   help for default-namespace

ds set server-url

Update the Dotscience Server URL

Usage:
  ds set server-url <url> [flags]

Flags:
  -h, --help   help for server-url

ds task

Manage tasks - control tasks

Usage:
  ds task [flags]
  ds task [command]

Available Commands:
  inspect     Display detailed information on one or more tasks
  ls          List tasks for the runner
  rm          Remove one or more tasks
  stop        Stop (terminate) one or more tasks

Flags:
  -h, --help   help for task

Use "ds task [command] --help" for more information about a command.

ds task inspect

Display detailed information on one or more tasks

Usage:
  ds task inspect [OPTIONS] TASK [TASK...] [flags]

Flags:
  -f, --format string   Format the output using the given Go template
  -h, --help            help for inspect

ds task ls

List tasks for the runner

Usage:
  ds task ls [OPTIONS] RUNNER [RUNNER...] [flags]

Aliases:
  ls, list

Flags:
      --format string   Pretty-print using a Go template
  -h, --help            help for ls
  -q, --quiet           Only display names

ds task rm

Remove one or more tasks.

Usage:
  ds task rm [OPTIONS] TASK [TASK...] [flags]

Aliases:
  rm, remove

Examples:

$ ds task rm my-task-uuid
my-task-uuid

Flags:
  -h, --help   help for rm

ds task stop

Stop one or more tasks. Tasks cannot be restarted.

Usage:
  ds task stop [OPTIONS] TASK [TASK...] [flags]

Aliases:
  stop, remove

Examples:

$ ds task stop task-uuid
task-uuid

Flags:
  -h, --help   help for stop

ds version

Display the version of the Dotscience command-line tool

Usage:
  ds version [flags]

Flags:
  -h, --help   help for version