Python Environments

Overview

This manual describes the best practices for working with Python environments in your SURF Research Cloud workspace. Using a dedicated Python environment ensures that your work is:

reproducible
portable across workspaces
independent of system-wide Python installation

Why Use Python Environments?

Python environment managers allow you to install exactly the same packages and versions that were used to develop a project, even on a new system such as a SURF Research Cloud workspace with minimal effort.

Without environment management, projects may:

break after system updates
behave differently across machines
be difficult to reproduce or share

If you would like to learn more about reproducible research workflows, the following workshop by Research Data Management at Utrecht University may be useful:

Best Practices in Writing Reproducible Code
Or read the chapter on dependency management

Python Environment Managers on SURF Research Cloud

This repository contains an example project for demonstrating how to work with different Python environment managing tools on SURF Research Cloud.

The repository provides information on working with the following workspaces:

The environment managers used are:

uv (recommended)
pyenv
poetry
conda

Information on how to create a workspace can be found here.

Getting Started

New to Python Environments?

Step 1: Identify Your Dependencies

Check if your project has one of these dependency files:

pyproject.toml
requirements.txt
environment.yml

If you do, move to Step 2.

Step 1b: Getting started with `uv`

If you are not working with an environment manager, we recommend using uv. Start by listing the packages you import in your code (e.g., pandas, numpy, matplotlib) and create a requirements.txt file with one package per line.

Installing uv is straightforward and fast.

Then initialize your environment in your project directory:

uv init 
uv venv --python 3.12  # or other version

Now activate it and install your dependencies:

source .venv/bin/activate
uv add -r requirements.txt

Step 1c: Capturing Dependencies

Alternatively, if you’re already working with an environment manager (uv, poetry, pyenv, conda) but don’t have a dependency file, you can use a command to capture your dependencies:

uv lock    # if you are working with uv

pip freeze > requirements.txt

conda export --file environment.yaml

Note: This captures everything in your current environment, so you may want to clean it up to include only what your project actually needs.

Step 2: Set up your environment on Surf Research Cloud

First make sure to move your files with your dependencies to your workspace. Good practice is manage your project using git, in that case you will need to push your files from your PC to github and clone your git project to your workspace. Otherwise, it is OK to transfer the files to your workspace using any of the data transfer methods.

On SURF Research Cloud, navigate to your project folder and create your environment:

uv sync  # if you have a pyproject.toml and uv.lock file
source .venv/bin/activate

If you have a requirements.txt file instead:

uv venv --python 3.12      # or other python version
source .venv/bin/activate
uv add -r requirements.txt  # install from requirements file
# OR
uv add pandas numpy matplotlib  # add packages individually

Start working:

python your_script.py

Using other Environment Managers?

conda, poetry, and pyenv are pre-installed, so you can continue using your existing workflow:

conda: Use your environment.yml file with conda env create -f environment.yml and activate with conda activate your-env-name.

venv/virtualenv: Create environments with python -m venv .venv and activate with source .venv/bin/activate. Install packages using pip install -r requirements.txt.

poetry: Continue using poetry install to set up your environment from pyproject.toml and poetry shell to activate it.