ResearchCloud Component Design

Mini-Workshop

Welcome!

Who are we?

ITS Research Engineering. Core activities include:

Projects
Trainings
Consultancy (e.g. HPC)
VRE service

uu.nl/rdm

Welcome!

Do you have experience with:

ResearchCloud?
Ansible / configuration management?
Docker / Podman?

Program

09.00: Walk-in
09.30: Welcome & intro to SRC
10.00: Some usecases for SRC components
10.15: Introduction to designing components & best practices
11.15: Design your own component!
12.15: Wrap up
12.30-13.00: Lunch

These Slides

You can find these slides here:

https://edu.nl/gy6np

ResearchCloud Lightning Intro

Some advantages of SURF ResearchCloud:

Easy access to compute and storage
Researchers can start VMs themselves
Easy collaboration (SRAM)
Cloud neutral: deploy VMs on SURF infrastructure or elsewhere

ResearchCloud Lightning Intro

SURF HPC cloud infrastructure:

up to 60 CPU cores
up to 750 GB RAM
up to 4 GPU drives (NVIDIA A10)
up to 1.5 TB storage

ResearchCloud Lightning Intro

Workspaces (aka VMs, servers)
- Running instances of a Catalog item
Catalog Items
- A set of components
- Parameters
- VM size
Components!
- (Installation) scripts
  - Ansible
  - Docker
  - Powershell

ResearchCloud Lightning Intro

ResearchCloud Portal

Documentation for ResearchCloud:

Our VRE Docs
SURF’s wiki

Some use cases for components

Adam El Kassimi (Elyra)
Dawa Ometto (Galaxy)

Creating components

Portal

Note about the special SURF components:

SRC-OS
SRC-CO
SRC-External (for different types)
SRC Nginx

SRC-External

Custom components are executed on the workspace from within SRC-External.

This can lead to complications due to:

Specific Ansible version on the workspace
Specific Ansible collections on the workspace
OS-dependencies on the workspace
…

Ansible

A Playbook is the main script file that will be applied on a machine managed by Ansible:

---
- name: Example component
  hosts: localhost # On ResearchCloud, the target host is always simply 'localhost'.
  tasks:

    - name: Output some text # Every task has a name
      ansible.builtin.debug: # Every task invokes some module
        msg: hello world # modules have arguments

# If you save this file to `test.yml`, you can run it with: ansible-playbook -i localhost, -c local -vv test.yml

Ansible

Fork the template repo
Clone your fork locally
cd to the repo directory

Ansible

Try running the test playbook on a container!

podman pull ghcr.io/utrechtuniversity/src-test-workspace:ubuntu_jammy

podman run -d --name src_component_test -v $(pwd):/etc/rsc/my_component ghcr.io/utrechtuniversity/src-test-workspace:ubuntu_jammy /sbin/init

podman exec src_component_test run_component.sh /etc/rsc/my_component/playbook.yml

Ansible

If you’re using Docker instead of Podman:

docker pull ghcr.io/utrechtuniversity/src-test-workspace:ubuntu_jammy

docker run --privileged -d --name src_component_test -v $(pwd):/etc/rsc/my_component ghcr.io/utrechtuniversity/src-test-workspace:ubuntu_jammy /sbin/init

docker exec src_component_test run_component.sh /etc/rsc/my_component/playbook.yml

Ansible

Modules
Variables
- ResearchCloud parameters will be available as variables in your playbook!
Conditionals
Loops

Ansible

- name: Example component
  hosts: localhost
  vars:
    foo: true
  tasks:

    - name: Output some text
      ansible.builtin.debug:
        msg: "{{ item }} is a number"
      with_items:
        - 1
        - 2
        - 3
      when: foo

Ansible

Roles and collections

An Ansible role is a reusable set of tasks that can be (re-)applied with distinct arguments.

An Ansible collection is a set of roles (as well as plugins and playbooks).

Both can be installed with ansible-galaxy.

Ansible

Roles

---
# Top of your playbook
- name: Example component
  roles:
    - role: my_local_role # from the 'roles' directory of your repo
    - role: uusrc.general.uv # from a collection
      vars:
        uv_venvs:
          - path: /tmp/test_venv
            python: 3.13
        uv_python_versions:
          - 3.11
  tasks:
    # You can also include a role from within your tasks
    - name: Include a role
      ansible.builtin.include_role: foobar

Ansible

Roles and collections

The dependencies declared in requirements.yml in the root of your repository will be automatically installed.

---
collections:
  - name: https://github.com/UtrechtUniversity/researchcloud-items.git
    type: git
    version: main
roles:
  - src: geerlingguy.docker
    version: 6.1.0

This is simply a standard requirements file for ansible-galaxy.

Ansible

Our collections

uusrc.general (main repo, docs)
uusrc.ibridges (iRODS data transfer)

Ansible

Don’t: use command and shell where you can use a dedicated module instead.

More portable
No need for error/output handling
ansible.builtin.shell especially can show unexpected behaviour

Examples: ansible.builtin.pip, ansible.builtin.service

Ansible

Don’t: depend uncritically on external roles.

Fix dependencies to specific tags or commits for stability and security.

Ansible

Do: process/sanitize incoming ResearchCloud parameters.

All parameters passed along by the ResearchCloud portal arrive in your playbook as strings:

Newlines are escaped
Possibly empty…

Ansible

Parsing incoming ResearchCloud parameters:

---
# Top of your playbook
- name: Example component parameters
  vars:
    _src_component_some_str: "{{ src_component_some_str | default('fallback value', true) | replace('\\n','\n') }}"
    _src_component_some_boolean: "{{ src_component_some_boolean | default(false, true) | bool }}"
    _src_component_some_dict: "{{ _src_component_some_dict | default({}, true) | from_yaml }}"
    # good luck with multiline yaml/json!
  tasks:
    ...

Ansible

Do: keep your playbooks simple.

Separate concerns into roles and task files:

roles for anything that has potential to be reused
separate tasks into files to prevent clutter

Ansible

Including task files:

ansible.builtin.include_tasks: tasks/dependencies/debian.yml
when: ansible_os_family == 'Debian'

ansible.builtin.include_tasks: tasks/dependencies/rh.yml
when: ansible_os_family == 'RedHat'

CI

The template repo contains boilerplate for:

molecule tests, configured especially for ResearchCloud
ansible-lint

Creating your own component

Build your own!

Suggested exercise: deploy a web application and add authentication.

Ideas:

ASReview
Ollama API
Theia IDE
Your own app!

See here for more detailed exercise instructions.

Think about the steps that need to be performed, e.g:

Create a python venv
Install ASReview in it
Run it as a system service
Add a reverse proxy

What parameters do you need?

Wrap up

How did it go?
Future steps