A Cloud of One's Own

I’ve been spoiled by the cloud. A decade and a half after I first used Dropbox, and years after iCloud made the dream of secure, seamless “login and forget” cloud sync a reality (most of the time), it seems obvious that all of my stuff should be available from every device I have whenever I need it. But what about content too big to keep on the cloud?

I’ve written before about how I went from a pastor to a media producer thanks to the pandemic. That’s only a slight exaggeration. When churches couldn’t meet in person, I went from someone who dabbled in video post-production and the like to someone who was generating literal terabytes of video to try to make the best use of the moment we were in. Post-pandemic such as we are (or hope to be), I still find myself keeping up with that media production habit.

With so much huge material, I dusted off the HP MicroServer I’d bought ten years prior, still running the ancient version of FreeNAS from when I first set it up, updated the software (now called TrueNAS Core) and replaced the drives to create a 12 TB storage array for all the media I was juggling.

The free, open source TrueNAS system running on any old desktop loaded up with several cheap, slow, but huge mechanical hard disk drives is a great way to get a lot of reliable storage for very little money (and, crucially, no reoccurring cost).

Since reviving my TrueNAS server, I’ve put video I purposefully film or edit for my work on the server when I’m done working with it. The one downside is that it isn’t accessible like Dropbox or iCloud when I’m away from the office. So, what do I do with the gargantuan files I still have to work on? Those found a spot in a folder on iCloud Drive, available even when I’m not in the office.

This less than ideal, but workable, arrangement served me well until my iCloud Drive filled up. Between documents I always keep in it, my large photo library and the other apps that use iCloud storage, a couple of months ago I started bumping into the upper limit of my 2 TB of cloud storage.

(Apple does make a bigger plan, but it involves a jump from $10/month to $30/month. That’s an added expense I’m unwilling to stomach.)

Perhaps you’ve hit a similar wall. There just has to be a solution, right?

A few weeks ago, I ran into an ad for Tailscale, a “mesh VPN” based on the open source Wireguard system. The idea is that you install a software program on your different systems and they form a peer-to-peer network able to communicate with each other securely even when they are not on the same physical network. Tailscale’s magic is a “coordination server” that helps the different systems you have form their initial link and become aware of additions to the system.

If it worked, Tailscale would allow me to go from depending on iCloud for files better suited to my network storage to being able to directly access that NAS. A great idea, but one with an immediate splash of cold water for me: Tailscale is not compatible directly with TrueNAS.

Trying to figure out how to make it work, I encountered Netmaker. Netmaker is an open source attempt to do what Tailscale does. Both use Wireguard and both draw from a lot of other open source projects as well. Netmaker, though, has a few nice differences:

1. I can install the “coordination server” — the part Tailscale provides free and paid subscriptions for on its end — on my own computer or cloud instance, avoiding dependency on a third party for coordinating my data. For all I know, Tailscale is the most trustworthy company ever, but I’d rather handle my data myself.

2. Unlike Tailscale, Netmaker allows one to connect the VPN to systems that only support its base, Wireguard, even if there is no direct client available. That means TrueNAS could connect to Netmaker despite lacking a Netmaker Client.

This all sounds good in theory, although getting the tool live for my use proved less than straightforward. For the not-faint-at-heart who are looking for an interesting challenge, though, the bliss of having my network available in my personal cloud is something you too can enjoy.

(To be clear: stop here and consider a paid service like Tailscale if you aren’t willing to delve into terminal commands some.)

The most straightforward Netmaker installation involves setting up a virtual private server (VPS) somewhere and installing Ubuntu Linux on it. I’ve used (and found good) both Ionos and Hivelocity VPS instances. If you follow me in this effort before the end of November, Hivelocity is the better deal, with their current 50% off promotion (coupon code “VPSBF50,” which is a fantastic deal, and one I get nothing for mentioning other than the warmth of knowing my readers are saving money).

Their smallest instance makes setting this up a no-brainer on the cost front with that coupon and appears to be powerful enough, at least for my needs. Add in a domain name that the server will live under a subdomain of — more than likely if I haven’t already scared you off, you already have a domain you could use — and you’re ready to begin.

First, go to your domain provider or host and set a wildcard A NAME. For example, if your domain is ofb.biz, and you want Netmaker to live at netmaker.ofb.biz, the A NAME would be *.netmaker.ofb.biz.

No one would give Netmaker credit for excellent documentation, but their quick start guide does a good job of getting the basics installed, if you simply enter the commands they tell you to on a VPS. Do what they instruct next.

(When prompted to install a license for the “professional” version, I opted out. My goal, after all, was to self-host and avoid being tied to third parties, not get myself more enmeshed with them and their fees.)

After running the Quick Install script, I got lost in the walkthrough before realizing the second follow-up link, appropriately entitled “Getting Started”, was what mere mortals wanting this to work need.

After the controller is installed, one goes to each computer you wish to have on Netmaker and installs the Netclient software. The previous step initialized a “dashboard” that is accessible on the web at dashboard.netmaker.ofb.biz (where “netmaker.ofb.biz” is replaced with the domain we configured earlier). The interface is reasonably user friendly and by logging in, going to the “Netmaker” network on the sidebar and clicking “Hosts” you can get to a page with a big “Add New Host” button that will walk you through not just installing the client software on a computer (node), starting by creating an “enrollment key” (think “invite code”) to allow the device you are adding to be added.

This is where I thought I finally had it made, but actually ran off the rails in a way that made me want to rip this in-the-works column to shreds and write about something else. (Except that it had driven me so insane, I couldn’t quit trying to get it to work.)

The instruction manual offers both a GUI (graphical) and terminal installation procedure for Macs. The web interface goes straight for that graphical install. Given the chance to have an administrative GUI, I thought that option made sense. I was wrong.

Depending on which link you use — the one in the documentation or the one in the web administrator interface — those running the latest MacOS will either get a package that is more up-to-date but incompatible with MacOS Sonoma or one that is compatible, but inexplicably several Netmaker client releases behind. Having run into the first, I went unknowingly to the second and thought things were good.

A small detour: my point in selecting Netmaker was primarily to make my NAS at my office accessible from home. If your NAS doesn’t natively support Netmaker, it’d be tempting to utilize Netmaker’s ability to accept “normal” Wireguard clients, but alas, those aren’t accessible to others on the VPN. A network storage server that is inaccessible is not useful.

Since TrueNAS doesn’t support Netmaker presently, I set up what is known as an egress server in the Netmaker dashboard (again under Networks > Netmaker). An egress server is just another Netmaker node/computer that happens to be on the same network as the devices you want to access remotely. I installed Ubuntu Linux on a tiny Mele mini-PC (affiliate link), followed the instructions for installing Netmaker’s Netclient on it and — voila! — my cloud VPS server could see and talk to anything on the office’s local network.

Things were not so charmed when I returned home to my MacBook Pro, which is where it is far more useful to access my network storage. It could not connect to anything else I put on the remote network for it to access (such as my NAS).

The fix proved simple: I completely uninstalled Netclient from my Mac and followed the alternate “brew” install instructions from Terminal.app. (If you don’t have Homebrew already installed, you need to get that first, using the one simple command on the Homebrew homepage. Installation takes a few minutes, but the tool will prove handy plenty of times outside of just installing Netmaker.)

The brew-based installation omits the GUI controls for Netmaker, but that’s no big loss. The graphical tool’s interface is too buggy to be helpful right now. The on/off toggle doesn’t work and the copy-and-paste step of adding a computer to Netmaker doesn’t work either with the app inexplicably preventing pasting.

Opening Terminal, I ran the following:

brew tap gravitl/netclient brew install netclient

This process will trigger a few MacOS security prompts that one can answer in the affirmative. At one point it offered to show me the installed file in Finder, I accepted and, right/two-finger clicked “Netclient” and clicked “open,” which let me assure MacOS I wanted to run this new app.

Next, I needed to get Netclient configured. From the aforementioned dashboard, I pretended to be adding a Linux system. I clicked the button to “add a new host,” clicked “Create New Enrollment Key” and followed the steps to create that invite code. Then, selecting that code, I told it I had a Linux system, ignored the instructions to download Netclient and clicked “next.” The subsequent screen displays a command to feed the Mac being added. I copied that command, returned to Terminal and ran:

sudo netclient install sudo netclient join -t [your key]

(Where the second line with “[your key]” is the command I copied from the web interface. Run the first command as is then paste the second one.)

This, unlike the GUI, brought my Mac to life on this handy Mesh VPN, with one caveat. Both the home and office networks have Eero routers and both assign computers IP addresses in the same private “range” by default, leading to possible conflicts. For the Eeros,these instructions let me change one of those — the one on my home network just to be safe — to a different range. (If your home and remote target networks likewise overlap in this way, check your router manufacturer’s information on how to customize one of those networks’ IP ranges.)

This last hitch and the solution to it are crucial given my egress server setup. By having my home network devices receive IP addresses in the 192.168.0.x range and my office network on the 192.168.4.x range, I can now access my office NAS from home by typing its same IP address as if I were right by it: 192.168.4.67. (Feel free to try that, it won’t work for accessing my NAS, unless you’re on my network!)

This may sound like an entirely too complicated project for a simple result like that. But simple is exactly why all the complexity is worth it, assuming it continues to work.

Now I can access files from (or transfer files to) my network storage even when I’m away. Ditto any other devices on the network. Even local screen sharing (such as VNC) works — I loaded Apple’s Screen Sharing app, typed in my office Mac mini’s IP and could use it with hardly any lag over the VPN. I even turned on its webcam and looked at the room with no perceptible delay when, for example, I used Alexa to turn on and off the lights.

All of the files I use in a cloud… under my own control. I’ve been waiting many, many years for this.