Python Environments

Overview

This manual describes the best practices for working with Python environments in your SURF Research Cloud workspace. Using a dedicated Python environment ensures that your work is:

  • reproducible
  • portable across workspaces
  • independent of system-wide Python installation

Why Use Python Environments?

Python environment managers allow you to install exactly the same packages and versions that were used to develop a project, even on a new system such as a SURF Research Cloud workspace with minimal effort.

Without environment management, projects may:

  • break after system updates
  • behave differently across machines
  • be difficult to reproduce or share

If you would like to learn more about reproducible research workflows, the following workshop by Research Data Management at Utrecht University may be useful:

Python Environment Managers on SURF Research Cloud

This repository contains an example project for demonstrating how to work with different Python environment managing tools on SURF Research Cloud.

The repository provides information on working with the following workspaces:

The environment managers used are:

Information on how to create a workspace can be found here.

Getting Started

To follow along with this manual, it is recommended to clone the repository mentioned above to your workspace. You can do this by running the following command in your workspace terminal:

cd <your-folder-name>  

git clone https://github.com/UtrechtUniversity/src-python-example.git # clone the repository

cd src-python-example # navigate to the project directory

New to Python Environments?

Step 1: Identify Your Dependencies

Check if your project has one of these dependency files:

  • pyproject.toml
  • requirements.txt
  • environment.yml

If you do, move to Step 1c

Step 1b: Getting started with uv

If you are not working with an environment manager, we recommend using uv. It is a modern, fast and user-friendly environment manager that is pre-installed on SURF Research Cloud workspaces. Installing uv is straightforward and fast.

First initialize your environment in your project directory with the python version you want to use:

uv init 
uv venv --python 3.12  # or other version

Now activate it and install your dependencies:

source .venv/bin/activate
uv add pandas # add packages to your environment
uv add 'pandas==2.0.3' 'numpy==2.0.0'  # if needed, you can specify package versions

Step 1c: Capturing Dependencies

If you have an environment manager (uv, poetry, pyenv, conda) set up, you can capture your dependencies in a file that can be used to recreate the same environment on SURF Research Cloud. This is important for reproducibility and portability of your project.

uv lock    # if you are working with uv

OR

pip freeze > requirements.txt

OR

conda export --file environment.yaml

Note: In the case of pip freeze, this captures everything in your current environment, so you may want to clean it up to include only what your project actually needs.

Step 2: Set up your environment on Surf Research Cloud

First make sure to move your files with your dependencies to your workspace.

Tip

Good practice is manage your project using git, in that case you will need to push your dependency file from your PC to github and clone your git project to your workspace.

Transfer your scripts and dependency files to your workspace using any of the data transfer methods.

On SURF Research Cloud, navigate to your project folder and create your environment:

uv sync  # if you have a pyproject.toml and uv.lock file
source .venv/bin/activate

If you have a requirements.txt file instead:

uv venv --python 3.12      # or other python version
source .venv/bin/activate
uv add -r requirements.txt  # install from requirements file
# OR
uv add pandas numpy matplotlib  # add packages individually

Start working:

python your_script.py # run your script (python main.py for example)

In a JupyterLab interface, you can register your environment as a Jupyter kernel so notebooks can use that environment to run code.

cd ~/src-python-example # Navigate to the project

source .venv/bin/activate # Make sure your environment is activated

# Register as Jupyter kernel (for src-python-example)
python -m ipykernel install --user --name=geo-kernel --display-name "geo-kernel (ipykernel)"

# For other projects
python -m ipykernel install --user --name {project-name} --display-name "Python ({project-name})"

# Verify the kernel is registered
jupyter kernelspec list

Refresh or close your JupyterLab tab. After this, you should see the new kernel in the JupyterLab interface when you create a new notebook or change the kernel of an existing notebook. Select the kernel with the name you specified (e.g., “geo-kernel (ipykernel)”) to run your jupter (.ipynb) notebook

Using other Environment Managers?

conda, poetry, and pyenv are pre-installed, so you can continue using your existing workflow:

conda: Use your environment.yml file with conda env create -f environment.yml and activate with conda activate your-env-name.

venv/virtualenv: Create environments with python -m venv .venv and activate with source .venv/bin/activate. Install packages using pip install -r requirements.txt.

poetry: Continue using poetry install to set up your environment from pyproject.toml and poetry shell to activate it.