Protogonus is our Beowulf cluster used for preparing, debugging and inspecting various parallel applications that use MPI.
This guide is provided as-is and we cannot be held liable for any outcomes or damages, either direct, consequential, or otherwise.
The following guide is an extract of our 2021 construction journal in an organised fashion. The end result is a Beowulf cluster using Ubuntu Server and Open MPI, with the cluster behind its own dedicated firewall and LAN, and the nodes using WIFI connection.
We assume that
— initially there is Windows on your machines, and that
— you will retain one of the computers as a support computer
The steps are more-or-less the same even if this is not the case.
To use the guide, it makes sense to
— read through the guide first, then
— read the journal, as that explains things in more details but more as the story of our discovery (including failures and dead ends), and then
— come back here to implement the steps of the guide
— your computers designated to be nodes (one will be the head node, and the others the workers)
— one support computer that will not be part of the cluster (you might do without this, if you do all your development on the head node, in which case the VPN steps of the guide can be omitted)
— an ethernet cable to eventually connect the computers to the network to download the wifi module
— a USB stick to be used as the installation medium
— extension cords for the power supply depending on your layout
— if you want to work without WIFI and use cables instead, the required cables and switches
Download to your support computer, the one you will not turn into a node (assuming Windows)
— Rufus for creating the bootable USB
— The latest version of the Ubuntu Server
Go through the wannabe node computers you have and
— make sure that you can connect to the WIFI from them (security of older network devices may be an issue)
— pick the most powerful computer. This will serve as the head. All the other computers will become workers.
Set up the Cluster LAN and the VPN, and make sure that
— you can reach the part of the network beyond the firewall
— you can reach the internet from beyond the firewall
Create the Ubuntu Server boot USB using Rufus. You should most likely use GPT in Rufus.
Come up with two usernames and passwords
— these will be the same on each of the computers
— use strong passwords,
— one username will be used as your system administrator during install
— the other one will be the user for the mpi system; also come up with a uid for this one, such as 911 or anything available below 1000; it needs to be the same across the cluster)
Design the addressing layout for your network
— Within the Cluster LAN, take note of your subnet and your gateway
— Decide on the fixed IP addresses: the head node could get IP 100, and the workers 100+i, i.e. 192.168.1.100 for the head node and 192.168.1.101 for the first worker, etc.
— Decide on hostnames for the nodes. node000 for the head and node101 for the first worker, etc. could be fine.
Ubuntu Server installation on each box one-by-one
Create, if needed, the install partition (using Windows)
— do not format and do not mount the newly created partitions
— one partition is relatively large, this will be the install target
— one is small (100 megabytes) for the boot partition
Boot the computer using the Ubuntu USB stick
— Getting this to work turned out to be the most annoying and most time-consuming step of them all.
— Do not plug in the network cable yet.
— You will most likely have to open the BIOS / UEFI settings, disable secure boot, disable OS optimized boot, and set the boot order so that the USB stick is used for booting. Opening the BIOS (UEFI) settings menu differs by manufacturers; you can also use a special Windows restart to reach the UEFI menu.
— If boot is successful, you will see the grub menu offering you to install Ubuntu Server
— make sure that the LAN cable is not plugged in yet
— do not use LVM
— do not use Open SSH; this will be installed later
— provide a proper username and proper (strong) password for the admin user. Use the same username and password for each computer. This is not the MPI user, that will come later
— if the installation is successful, follow the steps in the next section for the reboot
Reboot for the first time
— you can plug in the network cable now
— you might encounter some errors or timeouts while booting; these can be fixed later using our resources section
— log in with the admin username and password
Check to see if your internet is working OK
— keep the network cable plugged in
- sudo apt update
If there is a network problem, you will see it at this point
Set up WIFI on the box
If you do not wish to use WIFI, you can skip this step and keep running through the cable.
- sudo apt install network-manager
- sudo nmcli dev wifi connect <SSIDofYourWifiAccessPoint> password <YourWifiPassword>
— keep the network cable plugged in at this point
Set up the fixed IP of the node
- sudo nmcli con mod YourConnectionName ipv4.addresses xxx.xxx.xxx.xxx/24
- sudo nmcli con mod YourConnectionName ipv4.method "manual"
- sudo nmcli con mod YourConnectionNamd ipv4.dns "22.214.171.124 126.96.36.199"
- sudo nmcli con mod YourConnectionName ipv4.gateway YourDefaultGateway
It is important to use the ; between the two commands in the following row, so that you can reconnect through Putty (the connection will be lost either way). sudo reboot is also a possible solution.
- sudo nmcli con down YourConnectionName; sudo nmcli con up YourConnectionName
If the confirmation messages look OK, you can unplug the network cable and test the setup using
- ping google.com
Set up SSH
- sudo apt install openssh-server
- sudo ufw allow ssh
- sudo systemctl status ssh
The first ssh login will produce a warning message explaining that this connection is new. If you save the connection, this will not be repeated.
Install a few networking tools
- sudo apt-get install nmap
- sudo apt install net-tools
- sudo apt install clustershell
Install Open MPI (depending on worker or head node)
On the head node (steps might be redundant):
- sudo apt-get install libopenmpi-dev
- sudo apt-get install openmpi-bin
- sudo apt-get install openmpi-doc
- sudo apt-get install make
- sudo apt-get install g++mpirun
On the worker nodes:
- sudo apt-get install openmpi-bin
Add the mpiuser
- sudo adduser mpiuser --uid 911
You can choose any uid below 1000, not just 911. Since these are fresh and identical installs, the uid should be available everywhere.
Install NFS server depending on head or worker)
On the head node, use
- sudo apt-get install nfs-kernel-server
On worker nodes, use
- sudo apt-get install nfs-common
Set up the hostnames in the hostfile
You can use
- sudo nano /etc/hosts
Follow the instructions about the structure here . The most important is to make sure that only a single line contains localhost with 127… and all other lines reflect the actual IPs and hostnames of the cluster. In other words, the host file will be the same across the board.
Mark the node
You should put a sticker with the node count on the box and also to both ends of its cabling for less confusion later.
Phase One is complete
The remaining steps will be easier to implement once the nodes are set up in their own network
Optional step: wiping empty space
Depending on your layout, wiping empty space may be a next step. See details in the next section; however, making this step parallel does not always yield results, so we also placed this reminder here.
Repeat the above steps for all the desired nodes
Implementing the cluster
Arrange the nodes
— Arrange the nodes in their physical location
— Start the WIFI
— Start up all the computers
— From now on, all commands should be issued from the head node, unless stated otherwise
Check the network
List all the Ips of the nodes:
- sudo nmap -sP 192.168.1.0/24
or whatever your subnet is. If any worker is missing, check it.
Ping the master from all of the nodes:
- clush -w node[001-00n] -L ping -c 3 master
Here node001 – node00n should reflect your actual node hostname ranges. Clush can use  for ranges.
If there is any problem at this point, check it. You can use IP addresses instead of hostnames, or login to the nodes one-by-one using ssh.
Check to see if everything has access to the internet:
- ping -c 3 google.com
- clush -w node[001-00n] -L ping -c 3 google.com
This is important for updating Ubuntu even if you do not want the worker nodes to access the internet directly.
Optional step: wipe empty space
If you are using older computers, it may make sense to wipe their empty space, to make sure that you will not get into trouble due to GDPR or any other confidentiality reason at a later stage. Since this can be a time-consuming process, we can take advantage of the clustered setup and run all the wipes in one parallel. Based on your experience, directly logging in to the nodes might be the most stable solution, but you can also try ssh-ing to the nodes, or even clush.
A lot about this is described here .
- cat /dev/zero > zero.file
- rm zero.file
You can also use secure-delete’s sfill. It is not entirely clear if this makes any sense.
Set up passwordless login
The description is here and is fairly straightforward. We repeated the steps for both the admin user and the mpiuser.
Share the mpiuser folder using NFS
Either use the entire mpiuser folder as described below or login with the mpiuser, create a shareable subfolder, and use that (this might lead to less clutter and somewhat less flexibility)
to /etc/exports with the command
- sudo nano /etc/exports
- sudo service nfs-kernel-server restart
- sudo ufw allow from 192.168.1.0/24
In the last command, use your own subnet.
Check to see if it works on the workers:
- clush -w node[001-00n] -L showmount -e <IP of the head node>
Mount the mpiuser NFS folder on the workers
Anything before the clush command here is our workaround for the problem of entering the sudo password:
- echo <yourpassword> | clush -w node[001-00n] -L sudo -S mount <headnode>:/home/mpiuser /home/mpiuser
Test it out
- sudo touch /home/mpiuser/test
- clush -w node[001-00n] -L ls /home/mpiuser
On each worker, add the following line to /etc/fstab
<headnode>:/home/mpiuser /home/mpiuser nfs
Use ssh with the passwordless login and then nano; or perhaps copy an fstab file via the mounted mpiuser folder.
Use sudo reboot on the worker node (or if you managed to do it in bulk, you can use clush for the reboot)
Once the reboot is finished, your Beowulf cluster should be ready to do your HPC work
You can upgrade Ubuntu if you wish.