Install a library on your runner

Make your libraries available in Dotscience.

When you set up a Dotscience runner, you work inside a Docker container on that runner. This means that to use libraries in your model, it isn’t sufficient just to have them installed on the runner. They also need to be accessible inside the container.

Note that some common data science libraries (Panda, Numpy, Tensorflow) are installed in the container by default.

One-off installation

If you need to install a library that isn’t installed, the easiest way to do so is via the terminal on the JupyterLab web interface.

  1. Launch Jupyter, and select File > New > Terminal.
  2. Install your library. pip is available for installing Python packages. For example, you could run pip install flask.

Note that at present you will need to repeat this process every time you launch Jupyter.

Persistent installation

If you want to install lots of packages, and not have to reinstall them each time, there is another method you can use.

  1. Launch Jupyter, and select File > New > Text file from the menu.
  2. This text file should in requirements.txt format: one Python package per line. See below for an example.
  3. Choose File > Rename file… from the menu, and rename the file requirements.txt.

Keep in mind that unlike the default configuration, packages like Tensorflow will not be preinstalled, you’ll have to add them to requirements.txt yourself.

Here’s an example requirements.txt file:

tensorflow==2.1.0
numpy==1.18.1

Note: We recommend you pin particular package versions by using the package_name==1.2.3 syntax in order to ensure reproducibility.

To quickly test your requirements.txt for the first time, do the following:

  1. Launch Jupyter, and select File > New > Terminal.
  2. In the terminal, run pip download -r requirements.txt.
  3. If all goes well, the downloads will succeed, otherwise you can fix requirements.txt.

Now restart Jupyter:

  1. Click on the dotscience lab logo in the top right.
  2. Go to “Settings” tab for your Project.
  3. Click the “Stop Jupyter” button.
  4. Click the “Open Jupyter button to start it again.

On startup Jupyter will:

  1. Create a new virtualenv.
  2. Install the packages, as well as dotscience and ipykernel packages.
  3. Add a new Jupyter kernel called “Your custom Python 3”.

To use the environment that has your new packages, make sure to create a notebook or console using the “Your custom Python 3” kernel.

When installation fails

Sometimes installing new packages will fail. In that case, a file called ERRORS-FROM-REQUIREMENTS-INSTALL.txt will be created with the log of the installation, so you can debug the problem.

Fix your requirements.txt, then restart Jupyter.

You can test your requirements.txt by running ~/user-virtualenv/bin/pip install --upgrade -r ~/requirements.txt in the Jupyter terminal.