Adding Mergerfs To Your Synology

(Psssst…. TLDR?… fast forward to “The Implementation” below)

What is it?

mergerfs is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to mhddfsunionfs, and aufs.

Why would you do it?

Adding Mergerfs to your Synology could benefit you when you have multiple volumes with “the same type” data and you would like to have it accessed as a single mount point. For example on my Synology I have a SHR2 volume with 4 x 3TB disks summing up to 9TB (volume1) but I also have 1 single disk of 6TB (volume2) which I couldn’t (and didn’t want to) fit in volume1.

I have a DSM shared directory with some homevideos on volume1 (/video) and another directory with some more video on volume2 (/video2). I don’t want them all on volume1, so them divided in two shared directories. But I do want to them the be presented as one logical share to the outside world. Here Mergerfs comes in handy.

I would like to have /video and /video2 to be merged into a new logical shared folder called /videom (for example). The way Mergerfs does this is merging the two directory into one logical directory.

Why this page?

There is a clear wiki page explaining all this. But that setup comes with some issues (dated: 01-01-2022). 99% of the info is the way to go. In December 2021 I started to explore Mergerfs for my Synology. So I checked the wiki page from Mergerfs and I like the fact that I got an extra package manager (which is not maintained by the coder of mergerfs) with all the stuff Synology misses out on. But There are some problems.

The first problem is though that this package manager comes with a self-build version of Mergerfs that is not directly supported by Trapexit/Antonio (By the way Support Antonio if you can. His work is excellent on Mergerfs. I did support him too.). This version is at the moment: merge-with-ng-1131-gfe90b72b. Antiono helped me till the bitter end trying to get all of the below fixed, but he has no Synology and could not test much himself. So I was on my own. But I appreciated his help and efforts!!

The second problem is that the mergerfs mount command in the wiki mounts directly on the mointpoint of a shared directory made in DSM. This makes sense because you want the merged directories out in the world and it works on the first few looks of it, but when you start using it you soon run in more problems than you like. To name a few:

  1. The DSM reports that you are missing ACL on your shared directory after mergerfs mounts.
  2. When you would like to rename a file on the merged mount point, it does so with an error in the end.
  3. Hyperbackup will forever fail because it can’t read parts of the DSM config anymore (it seems Mergerfs fiddled with something on the share unintended)
  4. DSM also has a safety backup for the config, which backups to your Synology Account. This will fail too.

These are a few things I ran into in the first few days with the setup in the wiki. I started leaving Hyperback alone and started using Restic in a project I run on here and with a docker-image here. Did not like to be vendor locked by Synology on my backups anymore. But that’s out of scope for this blogpost. But the other stuff still was a problem.

The solution

After some sessions with Antiono to get issues fixes he couldn’t help me. 1. Mergerfs performed well on his systems. 2. The Synology was not the problem in itself either. What I understand from Antonio was that Synology made some custom implementations on they solution which would not be compatible with Mergerfs.

What I discovered was that mounting the merged directory not on a DSM shared directory but for instance on a mount point in /mnt like /mnt/videom would not have problems at all. But this mount point was not shared with the world and that was the whole point.

So I first tried to symlink this /mnt/videom to /volume1 which worked but I could not create the shared directory anymore and visa versa when I created the shared folder first I couldn’t symlink anymore. For the obvious reasons.

Then secondly I trying to mount mergerfs on a mount point within the shared directory. This worked, all the earlier problems were gone but Mergerfs only reports the maximum free disk space of the first branch, so I missed “half” the free disk space I would like to have.

The third try I used a mount -o bind on the directory in /mnt/videom and /volume1/videom, this was the Eureka moment! This fixed all problems but one. “The DSM reports that you are missing ACL on your shared directory after mergerfs mounts.” This is not a big problem since you set all config when creating the shared directory and not need to edit it anymore. But If you do, you can umount Mergerfs and edit it again. After that mounting Mergerfs again. I have understood that Synology has it’s own ACL implementation and Mergerfs is not compatible.

To get ride of the “unsupported” Mergerfs version from “Entware” I just got the correct version for my Synology from the releases page on GitHub. I guess this narrows down to the following builds for version 2.33.3 (date: 01-01-2022) but by the time you read this, the versions could have changed. I have a Synology DS1513+ with DSM 7.0. The DS1513+ has a Intel Atom processor. So I need to choose the “amd64” version from the pack. By the way, The custom build version from the Entware package manager works just fine, but when you get on the GitHub page and start issuing questions I know Antiono will tell you are not using an official release.

So for me all was fixed and I could happily use Mergerfs.

The implementation

SSH into you Synology box. And enter 'sudo su‘ to be superman for awhile. Be careful now you could mess up stuff in superuser mode.

  1. Create the directory where Entware will be installed. As said I like Entware as a package manager as it gives fast access to a lot of tool. I reckon you could get this to work without Entware, but I have not tried.
mkdir -p /volume1/@Entware/opt

2. Make sure we start fresh with an empty /opt directory. So remove it, recreate /opt and bind mount the this Entware

rm -rf /opt
mkdir /opt
mount -o bind "/volume1/@Entware/opt" /opt

3. Run install script depending on the processor. Use command ‘uname -m’ to find out. Then run the corresponding command.

armv8 (aarch64) – Realtek RTD129x

wget -O - http://bin.entware.net/aarch64-k3.10/installer/generic.sh | /bin/sh

armv5

wget -O - http://bin.entware.net/armv5sf-k3.2/installer/generic.sh | /bin/sh

armv7

wget -O - http://bin.entware.net/armv7sf-k3.2/installer/generic.sh | /bin/sh

x64

wget -O - http://bin.entware.net/x64-k3.2/installer/generic.sh | /bin/sh

4. Create the following scripts (init_entware.sh and init_mergerfs.sh) at a location of your liking.

init_entware.sh

#!/bin/sh

echo "*** Init Entware ***"

# Mount/Start Entware
mkdir -p /opt
mount -o bind "/volume1/@Entware/opt" /opt
/opt/etc/init.d/rc.unslung start

# Add Entware Profile in Global Profile
if grep  -qF  '/opt/etc/profile' /etc/profile; then
	echo "Confirmed: Entware Profile in Global Profile"
else
	echo "Adding: Entware Profile in Global Profile"
cat >> /etc/profile <<"EOF"

# Load Entware Profile
. /opt/etc/profile
EOF
fi

# Update Entware List
/opt/bin/opkg update

init_mergerfs.sh

Notes:

  • /mnt/videom is the mount point you need for Mergerfs; Set this to for likings.
  • /volume1/videom is the shared directory you need to create in DSM and this will get you out in the world
  • /volume1/video:/volume2/video2 are the branches you will to be merged. To my understand you can have more than two.
  • category.create=mfs is set to mfs, because I like the branches to be evenly filled. see here for more information
  • I use the nonempty option because I don’t want the Synology controle files (.DStore and @eaDir) to get in the way.
  • I added ‘modprobe fuse' before mergerfs so it mounts correctly
  • And of course the BIND Mount between /mnt/videom and /volume1/videom
#!/bin/sh

echo "*** Init Mergerfs ***"

# Check and create mountpoint in /mnt 
if [ ! -d "/mnt/videom" ]; then
    mkdir -p /mnt/videom/
fi

# Mount --bind the two directories
mount -o bind /mnt/videom/ /volume1/videom/

# Start Mergerfs
modprobe fuse
/opt/bin/mergerfs -o nonempty,rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=mfs,dropcacheonclose=true /volume1/video:/volume2/video2 /mnt/videom

5. Create One Autostart Task On Synology to init Entware

Notes: Edit the Run Command to reflex your setup on your Synology.

Create a triggered user-defined task in Task Scheduler.

Go to: DSM > Control Panel > Task Scheduler
Create > Triggered Task > User Defined Script
General
Task: Entware
User: root
Event: Boot-up
Pretask: none
Task Settings
Run Command: /volume1/beheer/scripts/entware/init_entware.sh

Notes: If you like (optional) you can set your logging directiory in the settings of the task manager.

6. Reboot your NAS

7. SSH back into your nas and ‘sudo su‘, enter in your password. Then run the following command.

opkg update

8. Install Mergerfs

opkg install mergerfs

9. Make sure it is installed

ls /volume1/@Entware/opt/sbin

10. Download and installed the latest version of Mergerfs from the Github releases. At time of this writing the version is 2.33.3. Download the correct file for you Synology system to your MAC or PC, uncompress it and copy the files (mergerfs and mergerfs-fusermount) from the archive in the directory /usr/local/bin to your Synology in /opt/bin (which is mount bind to /volume1/@Entware/opt/bin).

If things work out fine ‘mergerfs --version' will give you the current installed version from the Github releases.

Hint: rename the existing files, so when things don’t work out you can revert.

11. Create another Autostart Task On Synology for the Mergerfs to start when rebooted.

Notes: Edit the Run Command to reflex your setup on your Synology.

Create a triggered user-defined task in Task Scheduler.

Go to: DSM > Control Panel > Task Scheduler
Create > Triggered Task > User Defined Script
General
Task: Mergerfs
User: root
Event: Boot-up
Pretask: Entware
Task Settings
Run Command: /volume1/beheer/scripts/mergerfs/init_mergerfs.sh

12. Reboot your systeem and let the magic from Trapexit be executed!

Choosing your Python editing weapon

Well a few days passed since I got a micro:bit… Started out with MakeCode to play around with it, but the goal was python. For this I switched to mu-editor at first. A great editor… I like the check en tidy function. But I missed the simulator I had in MakeCode. Of course you can (and must) run your code on the physical micro:bit but a simulator is great for testing and trying little pieces of code. But I accepted the fact that mu-editor and python coding didn’t had a simulator.

Then I picked up a “old” tool from the shed on my Macbook, Visual Studio code. I started to use it again to edit shell scripts because brackets announced to stop service this coming summer. Why not pop in python script in there. This editor offered me to install python extensions. And I started exploring the nets of inter what was possible with VScode.

And I came across Device Simulator Express, an extension for VScode. Just the tool I needed to have my micro:bit simulated in VScode. Just open the extensions tab in VScode and look for “Device Simulator Express” and install the beast.

There is one pitfall in installing this extension. After installing the extension en pressing shift-command-p to open de command-pane : enter “device simulator express : New file”. This will install some more dependencies and you might run in the problem that you get a message “Device Simulator Express: Install Extension Dependencies”. Telling you the install failed.

When pressing shift-command-p and giving the command “Device Simulator Express: Install Extension Dependencies” as instructed, you will find that they will never get installed and you end up with a reoccurring error all the time. This seems to have something to do with python 3.9. You can fix this doing the following:

On Python 3.9:
Edit the file "\users\<user>\.vscode\extensions\ms-python.devicesimulatorexpress-2020.0.36321\out\requirements.txt" and changed the line

Pillow==7.0.0
to
Pillow==8.1.0

Restart VScode and press shift-command-p and give the command “Device Simulator Express: Install Extension Dependencies” again. Now the dependencies get installed nicely.

What I extra installed was a linter. Shift-command-p : Python: Select Linter and select “flake8”. Sometimes you need to restart VS-code to force install-popup for flake8. Now you have error-highting in the editor as you go.

And what I also installed was code formatter for python. Press shift-option-f: install formatter : install autopep8. This little baby formats your code. Cut the extra white-space. Correct the indenting. All kind of stuff.

I stopped using Windows at home for more than 11 years and left everything from Microsoft alone, switching to Apple and got the complete fruitbasket with all it’s products. But VScode has surprised me and it’s gonna stay for python editing……

Starbit Commander II

Well on my quest to learn python, I poked and peeked around on the web of inters. And came across some nice sites about the micro:bits, projects and coding.

One site inspired me to create a python version of Starbit Commander. This site is home to a lot of projects of Derek Graham. Next to Micro:bit bits, he has a lot of other stuff too on his site.

But why remake of Starbit Commander? Derek has this nice example of “Tiny Astroids”, which resembled a lot like Starbit Commander. And he created it in Python… sooo.. mine Starbit Commander should get a python version too… In this little gem from Derek, he created a really really nice piece of code that animated an explosion on this 5×5 led display. I loved it. And although I wanted to create Starbit from scratch in OO, I really wanted to adopt this nice little tiny explosion.. So I contacted Derek via twitter, asked if he agreed me borrowing his explosion-code. we had a short but very nice chat on twitter and he agreed for the code to be reuse in Starbit Commander. Thanks Derek!

Well, I wanted to test and learn more python and also OO in python. So I decided to try to make Starbit Commander in an OO-style. I have no experience in this but I think (as far as I can tell) this is an OO-version of the game now. Derek mentioned on micro:bit V1’s coding OO would let you ran into out of memory errors. I haven’t seen them on the microbit V2 with Starbit Commander. (Now I’m worried if I did a proper OO-coding style ;-0)

I differed a bit from the original Starbit code. While de MakeCode version has power-ups to collect, in this python-version I decided to skip that and make the astroidfield a bit more challenging by starting slow and easy and ending fast and astroid-crowded. Giving you more and more bonus score along the way you go.

This code can by found here. And watch that explosion… Thanks again Derek Graham!

Micro:bit Fireflies

My first attempt to write some (micro)Python code. Fireflies. This leds simulate fireflies in the air…

btw. I found that display.get_pixel does not return the correct value for the brightness of a led on a micro:bit. I created this function to correct this behavior in the code (fixPixelBug).

from microbit import *
import random

# This fucntions fixes the "display.get_pixel" bug.
# a full bright led does not return a "9" but a "255".
def fixPixelBug(brightness):
    if brightness == 4:
        return 3
    elif brightness == 8:
        return 4
    elif brightness == 16:
        return 5
    elif brightness == 32:
        return 6
    elif brightness == 64:
        return 7
    elif brightness == 128:
        return 8
    elif brightness == 255:
        return 9
    else:
        return brightness


while True:
    sleep(50)

    brightness = random.randint(1, 9)
    x = random.randint(0, 4)
    y = random.randint(0, 4)
    if display.get_pixel(x, y) == 0:
        display.set_pixel(x, y, brightness)

    for fireflies in range(0, 5):
        x = random.randint(0, 4)
        y = random.randint(0, 4)

        if fixPixelBug(display.get_pixel(x, y)) > 0:
            display.set_pixel(x, y, fixPixelBug(display.get_pixel(x, y)) - 1)

https://github.com/marc0janssen/microbit-fireflies/blob/main/fireflies%20v2.py

Simple Simon Says

The game Simon in the late 70’s was maybe the first “computer” game I played. Well maybe it was a real computer. I was just a kid, but the game stayed always with me. In my mind that is. The game was from a friend and we played it for ages.

Now with the Micro:bit I wanted to revive this memory and use the Micro:bit as a vehicle to get this to life again. Below is my make code attempt. I used an Micro:bit version2 for this.

This Simon listens to Buton A, Button B, Button C (the touch sensitive logo), Button D (= Button A+B).

It was fun to create and a good way to set me off with the possibilities of the Micro:bit…

To edit this repository in MakeCode.

Starbit Commander

This game is my first coding for The BBC Micro:bit. The goal is to have a nice target to practice and learn microPython on this device. But first I wanted to have a go on the makeCode editor for the Micro:bit.

I made this simple little game. Flying a spaceship through an astroidfield. Sometimes a blinking power-up will appear. This gives a the advance to survive an astroid collision. Catching two power-ups will destroy all astroids in the field. Astroids will speed up in time but slow down if a double power-up is acquired.

To edit this repository in MakeCode.

Pretty Good Privacy with keybase.io

I used PGP back in the 90’s just because it was possible. The internet was growing and my friends and I liked to experiment in those days with all that we found on the internet. PGP was one of those things. We had great fun back then but never used it again the following decades.

But a few days ago I saw a talk from Mike Godwin about privacy on the internet. He pointed out https://www.keybase.io/ in his talk as a start to set up PGP and ways to communicate with him.

I got curious again about PGP and Keybase.io and I had no trouble at all to quickly setup an account and a PGP key pair with these guys. They have nice low-level tooling to encrypt and decrypt messages on their website.

What I wanted again was a way to have my email encrypted, like I had back then in the 90’s when my friends and I played around with it. I found a great tutorial on the internet from the Electronic Frontier Foundation on how to het PGP setup on a MAC.

It is setup with Thunderbird Mail Client and within this tutorial they let you generate a PGP key pair with GnuPG. Which will do the job but I wanted to set it up with my Keybase.io key pair. I needed to export my Keybase.io key pair to the GnuPG keychain.

Reading the docs at their site, I found out that I could pull my Keybase.io Key to GnuPG keychain by the following command.

keybase pgp pull-private –all


But this gave me the following error

▶ ERROR .keys doesn’t exist

Error that .keys does not exist

Just following this workaround to fix it.

Make sure that in the linked device to your keybase.io account the option “Forbid account changes from the website” is disabled in the advanced settings. By disabling this option more possibles are enabled on the keybase.io site. One of them is to export your private key.

After you have disabled this option on your device, go to the website of keybase.io and visit your profile page. And find an “edit” link behind the signature of the public key. Select the edit link and you get the option to export your private key.

Exporting your private key

Copy the key and save it to your desktop. Use the following command to import the private key to the GnuPG keychain. Where “Private_Key.asc” holds your private key.

gpg2 –allow-secret-key-import –import Private_Key.asc

Also save your public key to your desktop. And import this one with the following command. Where “Public_Key.asc” holds your public key.

gpg2 –import Public_Key.asc

This serie of actions will replace the generation of a PGP key pair with GnuPG and import your keybase.io key pair.

DON’T FORGET TO DELETE YOUR PRIVATE KEY FROM YOUR DESKTOP.

Make sure it’s cleaned up!

Now finish the thunderbird tutorial from EFF with the keybase.io key pair. And you have a PGP mailclient setup with keybase.io. There is also a nice integration possible with the MacOS MAIL.APP with https://gpgtools.org but this requires a paid license. Whatever suites your needs.

Ok I hope this helps you.

Find me here on keybase.io and if you like to send me a PGP encrypted email, here is my public key. But this one can also be found on keybase.io.

Oh and don’t forget to send me your public key or your keybase.io profile page so I can download your public key if you shared it. If you want a message back of course ;-).

Use the following command pull in my public key in your GnuPG keychain. And a follow on keybase.io.

curl https://keybase.io/marc0janssen/pgp_keys.asc | gpg2 –import

keybase follow marc0janssen


Thanks for reading!