Dotscience Pipelines

Automation flows built with visual pipelines

Dotscience Pipelines

This is not a feature that is globally available on our site. If this is a feature you are interested in and we will enable this for your account, please reach out to us either on slack or contact us.

Note: this is a prototype feature, please expect some rough edges. We’re interested in your feedback. Let us know how you get on on Slack

Using Dotscience Pipelines, you can:

  1. Trigger a run
  2. See a status of whether the run is active, completed successfully or failed
  3. Process the logs/output the logs
  4. Trigger other useful services, such as slack or email

To use this feature, you will need a project with which to use Dotscience Pipelines. Create a project, and with an auto provisioned runner you are all set. If you wish to use your own runner, see the section on Runners

Hello World!

In this example, we will be running a basic python script from the command line, but in the real world, you will probably want to have already written a script which does something more interesting.

To start, go to the project you wish to use.

Click ML flows - this will take a little while to launch the UI on a runner.

Enter the username and API key for your account (should be shown at the top of the screen)

Select a trigger type from the widgets on the left - we could start with the timestamp one, this will allow us to trigger on click.

Select the dotscience script widget and link those two together - we can configure the login details and script we want to run later.

Select an end type - let’s use a debug logger. This will allow us to print the logs from the job.

To enter your dotscience details, double click on the dotscience script widget.

  1. The hostname should be your instance - so in this case, https://cloud.dotscience.com
  2. The username should be your username, API key can be found in Account -> API Key
  3. Enter the project you’re currently using for project
  4. For command, enter:

    python -c 'import dotscience as ds; ds.script(); ds.summary("count", 1); ds.publish("hello, world!")'

Finally, click “deploy flows” - note that if you don’t click deploy flows and then choose to leave this window or shut down the instance, your work will not be saved.

Click the trigger on the inject widget.

You should see:

  • The status of the widget changes to blue, and “running”

  • Once it changes to “completed”, click the debug panel to see the logs

  • If you go back to your project, you should see a shiny new run!