Quite some time has passed since my last tries to get a ceph cluster running on ARM and compiling it on 32-bit ARM. But with every unsolved problem, moles are not well known to forget about unfinished tunneling projects. It’s again time to blast the solid rock with some pieces of dynamite 🙂 Ah, wrong project… we are under water with ceph… So, we do dynamite fishing a quincy-octopus 😛 (do not get confused… octopus was a release of ceph, quincy v17 is the latest as of writing this)

I purchased an ODROID HC4 quite some weeks ago. It has a 64 bit ARM core and two slots for hard drives. And the drives I bought some years ago (4TB WD Red) are still not continuously used. They just served to keep some data temporarly.

As you may guessed already, installing ceph is still not an easy task. You can often not simply stick to the manual and this is, why I started writing again. In fact, I took two long way without success out of many more wrong ways, that ended very quickly. And here is the way, that was successful, with only minor headaches to crush.

Installation

After trying different things (original HC4, debian bullsey stable and unstable), I got over to try armbian for ODROID HC4. It is based on ubuntu and provides recent updates (the only containing ceph quincy aka v17.2). Flashing is easily done with Balena Etcher.

But to get it running, you need to get rid of petitboot, preinstalled on HC4 SPI flash. The instructions mentioned on the armbian page were not working. With latest kernel, the MTD devices do not show up any longer in the booted system (reached by holding the bottom button of HC4 and powering on) and also petitboot did not show up when a monitor was connected. I don’t know exactly why, but I soon took out the screws from the case and connected the UART. Fortunately, I had an original ODROID USB-serial converter at hand.

After power up, a minimal system presented itself on the console. So i issued the commands to erase the SPI flash:

$ flash_eraseall /dev/mtd0
$ flash_eraseall /dev/mtd1
$ flash_eraseall /dev/mtd2
$ flash_eraseall /dev/mtd3

This took (especially for /dev/mtd3) a few minutes… After re-powering the board, all went fine and the system came up. It made its SSH service available on the network. Default user should be root and password 1234. You then need to go through some little setup wizard and the system should greet you accordingly.

Now update the system, change the hostname and reboot:

$ apt update
$ apt upgrade -y
$ hostname <YOUR_HOSTNAME>
$ hostname > /etc/hostname
$ sed -i "s/odroidhc4/$(hostname)/" /etc/hosts
$ sed -i "s/^.*\(SIZE\)=.*$/\1=256M/" /etc/default/armbian-ramlog
$ reboot

For getting a nice status on OLED, you can easily install sys-oled-hc4 as a user with sudo permissions:

$ git clone https://github.com/rpardini/sys-oled-hc4
$ cd sys-oled-hc4
$ sudo ./install.sh

To bring a bit of color into life (and to quickly see, if you are root or somebody)

$ sudo curl --silent \
  -o /root/.bashrc \
  https://raw.githubusercontent.com/the78mole/the78mole-snippets/main/configs/.bashrc_root
$ curl --silent \
  -o ~/.bashrc \
  https://raw.githubusercontent.com/the78mole/the78mole-snippets/main/configs/.bashrc_user
$ sudo cp ~/.profile /root/

Installing Ceph

First step is to install cephadm. This tool helps a lot to bootstrap a ceph cluster.

$ sudo apt install podman catatonit cephadm ceph-common
$ sudo cephadm bootstrap \
  --mon-ip $(ip route get 1.1.1.1 | grep -oP 'src \K\S+')

You are done with the initial step… You can log in using the hostname and the credentials as shown. If localhost is shown as the URL, simply replace it by the IP or the hostname of your HC4.

Then log in to the web UI.

Now you are asked to provide a new password and re-login.

Now follow the step for „Expand Cluster“

For step 2, if the is no selection possible, you should check, if ceph has HDDs found that are appropriate. For this, you can simply ask it on command line:

root@ceph-erbw12-hc4-001:~# ceph orch device ls --refresh
HOST                 PATH       TYPE  DEVICE ID        SIZE  AVAIL.   REFRESHED
ceph-erbw12-hc4-001  /dev/sda   hdd   WDC_WD40EF[...]  4000G  Yes     11s ago...
ceph-erbw12-hc4-001  /dev/sdb   hdd   WDC_WD40EF[...]  4000G  Yes     11s ago...

Refresh only has effect for the next call, when applying changes, call it twice with 5 seconds pause between calls.

To prepare HDDs that have a partition table on it, you can scrub them with the following command.

$ sgdisk --zap-all /dev/sdb /dev/sda <...>

This should help, getting it prepared as OSDs.

When they show up with Yes in AVAILABLE column above, times up for switching back to the web UI. You then need to select a deplayment option (or keep the default) and add primary devices.

I took the model as a filter, so every HDD with the same model string on every HC4 get simply added when ceph is bootstrapping the host. Since we do not have SSDs or NVMes, it makes no sense to define WAL or DB devices… They are also not available here 🙂

We then can add services to this host. For the moment, we only add the MDS.

Next Review the settings and press Expand Cluster.

You should soon see someting happening in the Dashboard.

OK, that was the hardest part… I need to buy another HC4 and two hard disks of the same type and then I’ll continue this post 🙂

Troubleshooting

Ceph Node Diskspace Warning

This warning can be mostly ignored and it is not documented anywhere in the helt check documentation. The warning araises because Armbian is using a RAM-log (/var/log) that get’s rsynced to HDD (SD card on /var/log.hdd) every day. It is also rotated, compressed and purged on the card daily. This warning will usually be resolved automatically, especially with the 256M ramlog setting (40M was armbian default) and should not pop up to often or only after setting up the cluster, while a huge amount of loggin is created.

If the problem persists, you could dive into details using the healt check operation documentation.

Manager crashed

After one day of runtime, ceph GUI reported a crash of the manager daemon. To inspect this, you need the ceph command, which is included in ceph-common, we installed previously without a need at that time. But for administrative purposes, it is quite handy 🙂

To inspect the crash, we will first list all crashes (not only new ones):

$ ceph crash ls
## Alternative to show only new crashes
$ ceph crash ls-new

We will now get a detailes crash report.

In my case, I’m not sure, if this is just a side effect of the Healt-Warn state of the cluster, not being able to pull device metrics. We will see, if it persists 🙂

To get rid of the warning, just issue an archive command

$ ceph crash archive <ID>
# Or to archive all listed (not showing up in ls-new)
$ ceph crash archive-all

To delete older crashes (and also remove them from ceph crash ls), issue the following command.

$ ceph crash prune <OLDER_THAN_DAYS>
$ ceph crash prune 3                # Will remove crashes older than 3 days

Outtakes – History of False Tries 🙂

OK, it was a really hard way to get, where I’m now. But I don’t want to conceil the tunnels I digged that ended at a concrete wall…

Installation

Installed debian bullseye on your HC4 SD-Card (get an image from here) using e.g., Raspberry Pi Imager, or Balea Etcher (I’ve used the portable version). After putting the image on the SD-card, plug it into your HC4, power it up and wait for about 1 minute and re-power it. For inexplicable reasons sshd is not starting up the first time you plug it in with a freshly flashed image. The initial script (/root/firstboot.sh does simply not reboot wehen finished). But the second time, it works fine.

After the second boot, you will find your HC4 by pinging it with its local name bullseye-server.local or by looking into your routers client table.

>$ ping bullseye-server

Ping wird ausgeführt für bullseye-server.localdomain [172.22.3.58] mit 32 Bytes Daten:
Antwort von 172.22.3.58: Bytes=32 Zeit<1ms TTL=64
Antwort von 172.22.3.58: Bytes=32 Zeit<1ms TTL=64
Antwort von 172.22.3.58: Bytes=32 Zeit=1ms TTL=64
Antwort von 172.22.3.58: Bytes=32 Zeit<1ms TTL=64

Ping-Statistik für 172.22.3.58:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 0ms, Maximum = 1ms, Mittelwert = 0ms

You can then access the HC4 by SSH-ing into it and using odroid/odroid for authentication. After that, you should immediately change some default setting and the hostname, because we want to setup (many) more HC2s later. I prepared a script for it that can be accessed easily:

odroid@bullseye-server:~$ sudo apt update
odroid@bullseye-server:~$ sudo apt -y upgrade

odroid@bullseye-server:~$ sudo apt -y install curl
odroid@bullseye-server:~$ mkdir ~/bin
odroid@bullseye-server:~$ curl --silent \
  https://raw.githubusercontent.com/the78mole/the78mole-snippets/main/ceph/initial.sh \
  -o ~/bin/ceph_initial.sh
odroid@bullseye-server:~$ bin/ceph_initial.sh <NEW_HOSTNAME> <NEW_USER>

If you want to do all these steps manually, have a look in the ceph part of my GitHub snippets repo.

Be aware, that the system will use debian unstable, since the cephadm and other ceph packages are not available for arm64 in the stable repo.

After having this all setup correctly (on every node), you can install cephadm on your master node:

$ apt install cephadm

When everything was fine (expecially with the install.sh script, you can setup your monitoring node including the ceph dashboard. For that, run the following commands:

$ ip addr
$ cephadm bootstrap --mon-ip <IP_OF_END0_FROM_LAST_CMD>

When you finished this, you will get a hint, where to find the dashboard with given credentials.

Use the URL and credentials admin and the password shown to sign in to the dashboard.

Next, the ceph dashboard will ask you to change the password. Do it and keep the new one in mind (forever).

After sign in with the new password, ceph greets you with it’s welcome screen.

Also Graphana will e available to play with cluster data on port 3000 🙂

Adding OSDs – The Storage

To add OSDs in the dashboard, they first neew to be prepared. Look here for further information. I needed to clear them first from all dirt that I collected during the tests…

$ ceph orch devices ls

If your devices do not show up, remove all partitions from it, using fdisk e.g., fdisk /dev/sda and deleting all the partitions. To make the hdds even cleaner, issue the following command:

$ sgdisk --zap-all /dev/sdb /dev/sda <...>

This done, force a refresh of ceph devices

$ ceph orch devices ls --refresh

Wait a few seconds and issue the command again. You should see something similar:

There is also some text and a column REJECT REAOSON omitted by me due to space limitations… The YES in the AVAILABLE column is the most important information. If the drives show up here, they can be added to the ceph cluster via an OSD.

There is also another option to create a logical volume from one or more partitions or drives using pvcreate, vgcreate and lvcreate to make them fit a drive for ceph, but here it makes not much sense to introduce even more overhead.

Now you can add the devices to ceph by creating new OSDs in the dashboard

In the Advanced Mode accordeoan you can add primary devices by filtering. The result will look like this:

Now press the Preview button and accept the suggestion:

Press Create to continue.

Now you need to wait for a few minutes to see the OSDs show up.

The OLED does not yet work on bullseye unstable: Now update your repository cache and do an upgrade of the system. You should also change your timezone settings for the OLED of the HC4 to show the correct local time.

$ apt update
$ dpkg-reconfigure tzdata
$ apt upgrade

#### if using the ODROID HC4
$ apt install odroid-homecloud-display wget curl gpg jq
$ reboot

I struggeled a lot to install ceph on the ARM64 ODROID HC4… Here are my misleaded tries

$ mkdir -m 0755 -p /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/debian/gpg | \
   sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ echo \
    "deb [arch=$(dpkg --print-architecture) \
     signed-by=/etc/apt/keyrings/docker.gpg] \
     https://download.docker.com/linux/debian \
     $(lsb_release -cs) stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ apt update
$ apt-get install docker-ce docker-ce-cli containerd.io \
   docker-buildx-plugin docker-compose-plugin
$ sudo docker run hello-world

We need to build ceph ourself, because the packages do not contain many of the needed packages. Alternatively, you can run the management node on some x86 and only use arm64 for the OSDs.

$ git clone https://github.com/ceph/ceph.git
# or
$ git clone https://github.com/the78mole/ceph.git
$ git checkout quincy-release
$ ./install-deps.sh
$ cd ceph
# To prepare a release build... Takes some minutes
$ ./do_cmake.sh -DCMAKE_BUILD_TYPE=RelWithDebInfo
$ cd build
# Next step will take many hours (maybe some days)
$ ninja -j1

To be able to distribute the packages (we will need more than a single host for ceph to make any sense), we will setup a debian package repository. I will make mine public so you can skip the process of compiling your packages. I used a german tutorial on creating an own repository, a tutorial to host a package repository using GitHub Pages and PPA repo hosted on GitHub.

$ mkdir ~/bin && cd ~/bin
$ curl --silent --remote-name --location \
   https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm
#### or # wget https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm
$ chmod +x cephadm
$ cp cephadm /usr/sbin

#### On arm64, the cephadm package is not available, even if we have this 
#### python script already at hand. Therefore, we put it in /usr/sbin and
#### fake the package to be installed with equiv. Don't do this on other
#### non-ARM systems
#### vvvv Dirty Hack Start vvvv
$ apt install equivs
$ mkdir -p ~/equivs/build && cd ~/equivs
$ curl --silent --remote-name --location \
   https://raw.githubusercontent.com/the78mole/the78mole-snippets/main/ceph/cephadm-17.2.5-1.equiv
$ cd build
$ equivs-build ../cephadm-17.2.5-1.equiv
$ dpkg -i cephadm_17.2.5-1~bpo11+1_arm64.deb
#### ^^^^^ Dirty Hack End ^^^^^
#### If someone feels responsible to fix the real cephadm package for
#### all arches (it is a python tool !!!), please do it :-)

$ cephadm add-repo --release quincy
$ apt update
$ cephadm install
$ which cephadm   # should give /usr/sbin/cephadm

# Tweak needed for cephadm is to enable root login over SSH
$ sed -i \
  's/#PermitRootLogin.*$/PermitRootLogin yes/' \
  /etc/sshd_config
$ service ssh restart

Bootstrapping

First of all, here is a cephadm troubleshooting page.

After getting all in place, let’s start to setup the first host of the cluster.

$ ip addr
# Watch out for the ip of your main network interface and put it in place
$ cephadm bootstrap --mon-ip <IP_ADDR>

# If something fails, you need to stop the docker containers before retrying
$ docker ps
$ docker stop <THE_CEPH_CONTAINER_IDS>
$ cephadm ls | jq -r .[].fsid | sort -u
$ cephadm rm-cluster --force --fsid <ID_FROM_LAST_CMD>

To see, if everything is working correctly, we will restart the system.