Regulations

Governing Structure

The HPC@RUB cluster Elysium is operated by the HPC team at IT.SERVICES.

The governing structure of HPC@RUB is defined in the Terms of Use.

The HPC Advisory Board consists of five elected RUB scientists and two IT.SERVICES employees.

The five members of the current HPC Advisory Board, elected on April 18 2024, are:

• Prof. Ralf Drautz (speaker)
• Prof. Jörg Behler
• Prof. Sen Cheng
• Prof. Markus Stricker
• Prof. Andreas Vogel

FairShare

One of the main tasks of the HPC Advisory Board is to allocate a so-called FairShare of the HPC resources to Faculties, Research Centres, and Research Departments. Part of the FairShare is always reserved for scientists whose facility does not have its own allocated FairShare, so that the HPC resources are open to every scientist at RUB.

The FairShare is a percentage that determines how much of the resources is available to a given facility on average. A facility with a 10% FairShare can use 10% of the cluster 24/7 on average. If it uses less, others can make use of the free resources, and the priority of the facility to get the next job to run on the cluster will grow. If it uses more (because others don’t make full use of their FairShare), its priority will shrink accordingly. FairShare usage tracking decays over time, so that it is not possible to save up FairShare for nine months and then occupy the full cluster for a full month.

Within a given facility, all scientists that are HPC project managers share its FairShare. All HPC projects share the FairShare of their manager. Finally, all HPC users share the FairShare of their assigned project. This results in the FairShare tree that has become the standard way of managing HPC resources.

Project Management

HPC resources are managed based on projects to which individual users are assigned. The purpose of the projects is to keep an account of resource usage based on the FairShare of project managers within the FairShare of their facility.

Professors and group leaders can apply to become project managers; see the Terms of Use for details.

A project manager may apply for projects, and is responsible for compliance with all rules and regulations. Projects will be granted after a basic plausibility check; there is no review process, and access to resources is granted solely based on the FairShare principle, not based on competing project applications.

Users need to apply for access to the system, but access is only active if the user is currently assigned to at least one active project by a project manager.

Resources at RUB

HPC Cluster Elysium

Node Specifications

File Systems

The following file systems are available:

• /home: For your software and scripts. High availability, but no backup. Quota: 50 GB per user.
• /lustre: Parallel file system to use for your jobs. High availability, but no backup. Not for long term storage. Quotas: 1 TB and 1.000.000 files per user.
• /tmp: Fast storage on each node for temporary data. Limited in space, except for FatGPU nodes where multiple TB are available. Data is removed when the job ends.

Partition Overview

Two partitions are available for each type of compute node: the filler partitions are designed for short jobs, while the standard partitions support longer-running tasks.

Jobs in the filler partition have a lower priority and will only start if no job from the regular partition requests resources. Running jobs in the filler will cost only a fraction of the fair share of a regular partition.

The vis partition is special since the visualization nodes are intended for interactive use.

¹ Times of up to 7 days are possible on this partition but not recommended. Only 2 days are guaranteed, jobs running longer than that may get cancelled if that becomes necessary for important maintenance work.

² Cost does not refer to money, but the factor of computing time that is added to a projects used share in order to compute job priorities. The costs are based on the relative monetary costs of the underlying hardware.

³ Some of the memory is reserved for system services. Please check the scontrol show partition <partition_name> command to get the amount of memory that is available for your job via the --mem-per-cpu=<mem> submission flag.

Resources Elsewhere

HPC Pyramid

The HPC resources in Germany are arranged hierarchically in the so-called HPC pyramid.

If suitable for your needs, use the local resources provided by the tier-3 centre. If you need more resources than the local centre can provide, or your project requires specialized hardware, you are welcome to contact another HPC centre or request computing time at a higher tier (tier-2 or tier-1).

State-wide Computing Resources (Tier-2, Tier-3)

Several state-wide tier-2 centres (NHR centres) are available to cater for specialized computing and/or storage requirements. In North Rhine-Westphalia, the RTWH Aachen, the University of Cologne and the University of Paderborn offer structured access to HPC users from NRW institutes.

National and EU-wide HPC-Resources (Tier-1 and Tier-0)

For extremely complex and data-intensive requirements, HPC resources of the highest tier are available in Germany and the EU. Computing time is only allocated after a technical and scientific peer review process (GCS, PRACE).

Access to HPC Resources elsewhere

We would be happy to advise you on the suitability of your projects as well as provide help with the application process for all levels of the HPC pyramid. Please contact us.

HPC.NRW

The Ruhr-University of Bochum is part of the North Rhine-Westphalian Competence Network for High Performance Computing HPC.NRW. This network offers a first point of contact and central advisory hub with a broad knowledge base for HPC users in NRW. In addition, the tier-2 centres offer uniform, structured access for HPC users of all universities in NRW, ensuring basic services are provided for locations without tier-3 centres and for Universities of Applied Sciences.

A network of thematic clusters for low-threshold training, consulting and coaching services has been created within the framework of HPC.NRW. The aim is to make effective and efficient use of high-performance computing and storage facilities and to support scientific researchers of all levels. The existing resources and services that the state has to offer are also presented in a transparent way.

Get User Access

In order to get a user account on Elysium you need to download and fill out the user access application form. Note that you can only login to the cluster after you were assigned to at least one project by a project manager.

RUB LoginID: This is your RUB-LoginID. You need to active two-factor authentication for it.

SSH Key (Pub): Your public (not private!) SSH Key. This key must be your own. Sharing keys with others, e.g. members of your work group is not allowed. You can generate an SSH key pair with the following command:

ssh-keygen -t ed25519 -f ~/.ssh/elysium -N "passphrase"

Then you enter the contents of the file ~/.ssh/elysium.pub in the field. The “passphrase” should of course be changed to an appropriately complex password, that prevents malicious usage of your key. Note that you should only do this once. Running the command again, will overwrite your key and block you from accessing the cluster.

Note that RSA keys must have at least 3000 bits in accordance with BSI regulations. We recommand ED25519 keys, as shown in the example above.

After you correctly filled out the form save the PDF as User-Access-Application_<loginID>.pdf e.g. User-Access-Application_mamuster.pdf and send it via email to hpc+applications@ruhr-uni-bochum.de

Updating the SSH-Key

If you want to register another ssh-key or you lost your already registered one, you simply fill out the User-Access-Application again, and send it in with the exact same filename. If your ssh-key was compromised or stolen please notify us immediately so that we can invalidate it.

Apply for a HPC Project

After you were approved as a Project Manager you can apply for research projects on Elysium by downloading and filling out the project application form. The application is required for managing the fair share of computing resources, and for reporting to the HPC-Beirat, Funding organizations, etc. There is no peer-review process and your project is automatically accepted. Note that more projects do not mean more computing resources. Your personal resources are shared between all you projects.

RUB LoginID: This is your RUB-LoginID.

Project Name: Name under which your project should be listed in any report.

Abstract: A short 2-3 line abstract outlining the contents of your research project.

Field of Science: Identification number according to DFG subject classification system.

Third-party funded: Optional field for research projects with third party funding. E.g. Funding institution, or project number

RUB LoginIDs: Comma separated list of RUB-LoginIDs of the people you want to participate in this project. (Note that the project manager’s LoginID is not automatically included in the participant list. If you, as a project manager, want to participate in your own project, you must enter your Login ID as well.)

Contingent: From the dropdown menu select the computing resources contingent of your project. If you did not participated in the HPC cluster application you need to select “Miscellaneous User Groups”, or get permission by the members of the other contingents to use their resources.

After you correctly filled out the form save the PDF as Project-Application_<loginID>_<number>.pdf e.g. Project-Application_mamuster_5.pdf. The numbering of projects starts at 0 due to technical reasons. Your first project will have number 0, your second will have number 1, … . In the example given above the number 5 refers to the sixths project application that was send in by you, and send it via email to hpc+applications@ruhr-uni-bochum.de

Updating a Project

If you need to add or remove users you can simply modify the user list in the pdf, and send it in again with the exact same filename.

Become Project Manager

In order to get computing resources on Elysium and be able to manage projects you need to download and fill out the project manager application form and sign the compliance to export control regulations form.

Please note that only professors and independent group leaders within the Ruhr-University are eligible to becoming project managers! See the regulations for details.

Group Name: The name of your work group. E.g. “Chair for constructive demolition techniques”, or “Computatinal analysis of sumerian poetry”,…

Faculty/Institute: The name of the faculty your group is located in. E.g. “Faculty of Mathematics”, “ICAMS”, or “Universitätsklinikum Josefs-Hospital”

RUB LoginID: This is your RUB-LoginID.

Email: Your RUB-email. (Not your institute email address!) e.g. max.muster@ruhr-uni-bochum.de

Signed export regulations: After you read, understood, and signed the compliance to export control regulations form linked above, you check this box.

After you correctly filled out the form, save the PDF as Projectmanager-Application_<loginID>.pdf e.g. Resources-Application_mamuster.pdf. Save a scan of the signed export control regulations form as Compliance-Export-Control-Regulations_<loginID>.pdf e.g. Compliance-Export-Control-Regulations_mamuster.pdf. Send the application form and a scan of your signed export control regulations form via e-mail to hpc+applications@ruhr-uni-bochum.de

Basics

Elysium provides four login nodes: login1.elysium.hpc.rub.de, …, login4.elysium.hpc.rub.de. These are your entry points to the cluster.

After login, you typically use them to prepare your software and copy your data to the appropriate locations.

You can then allocate resources on the cluster using the Slurm workload manager.

After submitting your request, Slurm will grant you the resources as soon as they are free and your priority is higher than the priority of other jobs that might be waiting for some of the same resources.

Your priority depends on your waiting time and your remaining FairShare.

Login

Login to Elysium combines the common SSH key-based authentication with web-based two-factor authentication. In order to be able to authenticate during login you need submit your public SSH key via the User Access Application Form as well as enable two-factor authentication for your RUB LoginID at rub.de/login.

The additional web-based authentication is cached for 14 hours so that you typically only have to do it once per work day, per login node, and per IP address you connect from. After that, your normal key-based SSH workflow will work as expected.

In order to simplify the use of SSH keys we recommend to specify it as identity file in your SSH config. This can be done by adding the following lines to your ~/.ssh/config file:

Host login00*.elysium.hpc.rub.de login00*.elysium.hpc.ruhr-uni-bochum.de
    IdentityFile ~/.ssh/elysium
    User <loginID>

where <loginID> has to be exchanged by your rub-loginID. If your SSH key is located in a different file the IdentityFile path needs to be adjusted accordingly.

Follow these steps:

1. Start ssh with the correct private key, your RUB LoginID, and one of the four login hosts, e.g.
   ssh -i ~/.ssh/elysium LOGINID@login001.elysium.hpc.ruhr-uni-bochum.de, or ssh login001.elysium.hpc.rub.de if you want to use the SSH config specified above. Available login nodes are login001 to login004.

2. Open the URL in a browser (or scan the QR code with your smartphone) to start web-based two-factor authentication.

3. Enter the second factor for two-factor authentication.

4. After successful login, you get a four-digit verification code.

5. Enter this code at your ssh prompt to finish login.

For the next 14 hours, only step 1 (classic key-based authentication) will be necessary on the chosen login node for the IP address you connected from.

Login will fail if:

• You use the wrong private key (“Permission denied (publickey)”)
• You are not member of an active HPC project (“Permission denied (publickey)”)
• You did not enable two-factor authentication for your LoginID (“Two-factor authentication is required”)
• Web-based login fails
• You enter the wrong verification code (“Verification failed”)
• A timeout happens between starting the SSH session and finalizing web-based login (“session_id not found”); just start the process again to get a new session ID.

Software

We provide a basic set of toolchains and some common libraries via modules.

To build common HPC software packages, we provide a central installation of the Spack package manager. A detailed guide on how to use this installation can be found in the Spack Usage Guide.

SLURM

The Elysium HPC system utilizes SLURM as a resource manager, scheduler, and accountant in order to guarantee fair share of the computing resources.

If you are looking for technical details regarding the usage and underlying mechanisms of SLURM we recommend participating in the Introduction to HPC training course.

Examples of job scripts for different job types that are tailored to the Elysium cluster can be found in the Training Section.

List of Partition

All nodes in the Elysium cluster are grouped by their hardware kind, and job submission type. This way users can request specific computing hardware, and multi node jobs are guaranteed to run on nodes with the same setup.

In order to get a list of the available partitions, their current state, and available nodes, the sinfo command can be used.

 1[login_id@login001 ~]$ sinfo
 2PARTITION      AVAIL  TIMELIMIT  NODES  STATE NODELIST
 3cpu               up 7-00:00:00      4  alloc cpu[033-034,037-038]
 4cpu               up 7-00:00:00    280   idle cpu[001-032,035-036,039-284]
 5cpu_filler        up    3:00:00      4  alloc cpu[033-034,037-038]
 6cpu_filler        up    3:00:00    280   idle cpu[001-032,035-036,039-284]
 7fat_cpu           up 2-00:00:00     13   idle fatcpu[001-013]
 8fat_cpu_filler    up    3:00:00     13   idle fatcpu[001-013]
 9gpu               up 2-00:00:00     20   idle gpu[001-020]
10gpu_filler        up    1:00:00     20   idle gpu[001-020]
11fat_gpu           up 2-00:00:00      1 drain* fatgpu005
12fat_gpu           up 2-00:00:00      5    mix fatgpu[001,003-004,006-007]
13fat_gpu           up 2-00:00:00      1   idle fatgpu002
14fat_gpu_filler    up    1:00:00      1 drain* fatgpu005
15fat_gpu_filler    up    1:00:00      5    mix fatgpu[001,003-004,006-007]
16fat_gpu_filler    up    1:00:00      1   idle fatgpu002
17vis               up 1-00:00:00      3   idle vis[001-003]

Requesting Nodes of a Partition

SLURM provides two commands to request resources. srun is used to start an interactive session.

1[login_id@login001 ~]$ srun -N 1 --partition=cpu --job-name=test --time=00:05:00 --account=testproj_0000 --pty bash
2[login_id@cpu001 ~]$

sbatch is used to request resources that will execute a job script.

1[login_id@login001 ~]$ sbatch -N 1 --partition=cpu --job-name=test --time=00:05:00 --account=testproj_0000 myscript.sh
2Submitted batch job 10290

For sbatch the submission flags can also be incorporated into the job script itself. More information about job scripts, and the required and some optional flags can be found in the Training/SLURM Header section.

On Elysium several flags are mandatory. sbatch and srun will refuse to queue the job and give a detailed explanation which flag is missing and how to incorporate it into your command or script.

Use spredict myscript.sh to estimate the start time of your job.

Shared Nodes

All nodes are shared by default. If a user requests fewer CPU-cores than a node provides, other users may use these resources at the same time. To ensure that the requested nodes are not shared use the --exclusive flag. If more than one node is requested the --exlusive flag is mandatory.

GPU Nodes

For requesting resources on a GPU node the --gpus=<number of GPUs> flag is required. In order to allow for fairly shared resources the number of CPUs per GPU is limited. Thus the --cpus-per-gpu=<number of CPU cores per GPU> is required as well. For multi node jobs --gpus-per-node=<number of GPUs per node> option needs to be set.

Visualization Nodes

For requesting resources on a visualization node no --gpu parameter is needed. The available GPU will automatically be shared between all jobs on the node.

List of Currently Running and Pending Jobs

If requested resources are currently not available, jobs are queued and will start as soon as the resources are available again. To check which jobs are currently running, and which ones are pending and for what reason the squeue command can be used. For privacy reasons only the user’s own jobs are displayed.

1[login_id@login001 ~]$ squeue
2             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
3             10290       cpu     test login_id  R       2:51      1 cpu001

List of Computing Resources Share

Users/Projects/Groups/Institutes are billed for computing resources used. To check how many resources a user is entitled to and how many they have already used the sshare command is used. For privacy reasons only the user’s own shares are displayed.

1[login_id@login001 ~]$ sshare
2Account                    User  RawShares  NormShares    RawUsage  EffectvUsage  FairShare
3-------------------- ---------- ---------- ----------- ----------- ------------- ----------
4testproj_0000          login_id       1000    0.166667    20450435      0.163985   0.681818

List of Project Accounts

Due to technical reasons the project names on Elysium have rather cryptic names, based on the loginID of the project manager and a number. In order to make it easier to select a project account for the --account flag for srun, or sbatch, and to check the share and usage of projects, the RUB-exclusive rub-acclist command can be used.

1[login_id@login001 ~]$ rub-acclist
2Project ID    | Project Description
3--------------+--------------------------------------------------
4testproj_0000 | The fundamental interconnectedness of all things
5testproj_0001 | The translated quaternion for optimal pivoting

Visualization

We provide Visualization via VirtualGL on the visualization nodes on Elysium.hpc.ruhr-uni-bochum.de

Requirements:

X11 server with 24-bit- or 32-bit Visuals. VirtualGL version > 3.0.2 installed.

You can check support for your Operating Sytsem at: https://virtualgl.org/Documentation/OSSupport You can download VirtualGL at: https://github.com/VirtualGL/virtualgl/releases

To use VirtualGL on Elysium, you will only need the VirtualGL client, it is not necessary to configure a VirtualGL Server.

Resource allocation:

Allocate resources in the vis partition.

salloc -p vis -N1 --time=02:00:00 --account=$ACCOUNT

This will allocate a share of one vis node for 2 hours. (For more options on node allocations see SLURM). Wait until a Slot in the vis partition is available. You can check if your resources are already allocated using the ‘squeue’ command.

Establish Virtual GL connection:

Connect directly from your computer to the visualization node via ssh with vglonnect -s Use one of the login servers login[001-004] as a jump host.

vglconnect -s $HPCUSER@vis001.elysium.hpc.rub.de -J $HPCUSER@login001.elysium.hpc.rub.de

If you don’t like long commands, you can configure one of the login nodes as jump host in your ~/.ssh/config for the vis[001-003] hosts. The command vglconnect -s accepts nearly the same syntax as ssh.

Run your Software:

Load a module if required. Start your application using vglrun, please remember to use useful command line options like -fps .

module load vmd
vglrun +pr -fps 60 vmd

Please remember to cancel the resource allocation once you are done with your interactive session.

scancel $jobID

Job Monitoring

With our web-based job monitoring system (ClusterCockpit), you can easily monitor and analyze the performance of your jobs on the Elysium HPC system. For a quick performance check, see Metrics to Check; for an in-depth analysis, refer to the HPC-Wiki. For details on the web interface, consult the official documentation.

Login

To access the job monitoring system, use your RUB LoginID and corresponding password as credentials.

Overview

After logging in successfully, you will see the “Clusters” overview, which displays the total number of jobs you have run and the current number of jobs running on the cluster. At present, this information includes only the Elysium cluster. You can continue from here, either by going to the total jobs overview, or the running jobs overview. Alternatively, you can click on “My Jobs” in the top left of the page, or search for job names/ids in the top right of the page.

My Jobs

The “My Jobs” page displays a list of your jobs, fully customizable to your requirements. Use the menus in the top left corner to sort or filter the list, and select the metrics you want to display for your jobs. Below, you’ll find a detailed table with job IDs, names, and your selected metrics.

Job Details

This page is split into three sections. The first one shows general information: JobInfo, a footprint and a roofline diagram that shows how efficiently the job utilized the hardware. Note that the footprint is only updated every 10 minutes and the energy footprint is generated after the job finished.

In the next section some metrics are shown as diagrams. For some of the diagrams you can choose the scope, i.e. core, socket or node. The shown metrics and their order can be customized with the “Select Metrics” menu. This selection is saved per partition. Double-click the graph to zoom out if the scale is too small.

The last section displays selected metrics in a numerical way, lets you inspect your job script, and shows more detail about the job allocation an runtime parameters.

Metrics

The following table shows the metrics which are available for jobs on Elysium:

Metrics to Check

For a quick performance analysis, here are some key metrics to review:

• cpu_user: Should be close to 100%. Lower values indicate system processes are using some of your resources.
• flops_any: Measures calculations per second. On Elysium, a typical CPU node averages around 400 GFLOPS.
• cpu_load_core: Should be 1 at most for non-OpenMP jobs. Higher values suggest oversubscription.
• ipc: Instructions executed per cycle. Higher values indicate better efficiency.
• mem_bw: Memory bandwidth, maxing out at 350 GByte/s. Only meaningful if the node isn’t shared or your job uses a full socket.
• acc_utilization: GPU compute utilization. Aim for high percentages (e.g., above 80%) to ensure efficient GPU usage.

Known Problems

Occasionally, an orange box labeled “No dataset returned for <metric>” may be shown instead of the graph. This occurs when the ClusterCockpit service was unable to collect the metrics during your job. Note that jobs that ran before March 12th 2025 may report missing or incorrect data in some cases.

The measurements for ipc and clock are sometimes too high. This is related to power saving features of the CPU. We are currently investigating how to solve this issue.

For jobs that ran before March 7th 2025 a bug triggered an overflow for the power usage metric resulting in unrealisticly high power consumptions. This bug is fixed, but the fix cannot be applied to older jobs that were affected by it.