Bare metal home server

Why bare metal? What's wrong with the cloud?

There is literally nothing wrong with the cloud.

In fact, the cloud is better than bare metal on almost every front, even the historic ones; it's got a wider variety of offers at competing prices, no upfront costs or depreciating assets, it's highly available (gone are the dial-up days, enter the mobile broaband days) and it's often more secure as it's not up to you to stay on top of software updates, vulnerabilites patching etc.

For me personally, it was a mix of attempting to save some money (we were spending 20..50£ pcm at times), privacy concerns and, last but definitely not least, personal old-school professional interest in what the FOSS world had to offer.

But it does require some upfront expenses, a decent upload speed, runnig cost of electricity and more importantly, time and effort to set it up and maintain it.

Hardware

The choice obviously depends on your hosting needs. I decided to go for something modest for my main server and a high power single board computer for CPU-intensive tasks, so I can switch if off when I don't need it.

Main server:

• an AMD E350 motherboard/CPU (1.6 GHz, 2 cores 2 threads)
• onboard 1 GBbit ethernet
• 16 GB of RAM (2x8 GB)
• a cheap 128 GB SSD for the OS and configuration
• a PCI express SATA/RAID controller
• 4x 4 TB SATA HDDs in RAID0+1, for a total of 8 TB of storage with 2 disks-redundancy; it performs very well, with a read/write throughput over ethernet of about 60 MBytes/s
• Power-wise: about 50 W, i.e. 1.2 KW/h per day

Jenkins workhorse:

• Raspberry Pi 5

I'm not going to post links and prices as you are better off searching for the best offer of the day (Google, ebuyer, eBay, Amazon, AliEpxress) but it's definitely in the "affordable" (300-500£) range altogether, especially if you couple it with a 2nd hand PC cabinet and PSU.

Software

It runs:

The jenkins workhorse machine setup is covered in the Jenkins nodes article section.

Operating system

I chose Ubuntu server for its ease of use but I've used Debian and Arch in the past and can testify to their lighter weight.

You can go for the minimal installation and then manually install service by service from the above table, including docker (see below).

Auto-completion stopped working for me at some point and I found out bash-completion is now a separate package:

sudo apt install bash-completion

RAID

<TBD>

Periodically check your drives

Install smartmontools and run:

smartctl -a

Replace a failed drive

The MD server will automatically remove failed drives but it in case the drive is not completely dead or "faulty enough", remove it manually first:

sudo mdadm --manage /dev/md2 --fail /dev/sdc1
sudo mdadm --manage /dev/md2 --remove /dev/sdc1

Then replace it. You'll need to find the serial number, don't rely on any "original" /devsdX mapping (unless you set up some strict udev rules); it's at the top of smartctl -a /dev/<drive>

The partition the new drive using an existing drive as template:

sudo sfdisk -d /dev/<GOOD DISK> | sudo sfdisk /dev/<VIRGIN DISK>

Now add the partitions to the RAID. E.g.:

$ sudo mdadm --manage /dev/md127 --add /dev/sdc3

$ cat /proc/mdstat

Personalities : [raid10] [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : inactive sdb2[2](S) sdd2[1](S) sdf2[0](S)
      1566720 blocks super 1.2

md0 : inactive sdb1[2](S) sdd1[1](S) sdf1[0](S)
      12570624 blocks super 1.2

md127 : active raid10 sdc3[4] sdd3[2] sdb3[1] sdf3[3]
      7804335616 blocks super 1.2 64K chunks 2 near-copies [4/3] [_UUU]
      [>....................]  recovery =  0.3% (13014656/3902167808) finish=471.6min speed=137418K/sec

Docker

You can install docker with the following:

sudo apt install docker.io

and make your admin a docker operator so you can just "docker" instead of "sudo docker" everything:

sudo usermod -a -G docker <DOCKER ADMIN ACCOUNT>

Next we can configure the docker service to house docker data (images, containers..) somewhere on the RAID drive rather than /var/lib/docker, and allow using our local docker registry "insecurely" over LAN:

sudo nano /etc/docker/daemon.json

{
        "data-root": "<RAID drivepath>/docker",
        "insecure-registries":["<SERVER IP>:<Docker registry port>"]
}

MySQL server

I used it for my full stack applications, not mandatory.

sudo apt install mysql-server

TODO: add admin user

By default, the MySQL server will listen on port 3306 but only bind to the "localhost" IP, 127.0.0.1, making it impossible for docker containers and LAN hosts to connect, so you'll have to configure it accordingly:

sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

bind-address            = 192.168.0.100
mysqlx-bind-address     = 192.168.0.100

MiniDLNA (media browsing/streaming)

This allows to share multimedia files with digital players, smart TVs and software like VLC or Kodi (check out Libreelec too).

sudo apt install minidlna

and then list your directories by content (P=pictures, V=video, A=music) in /etc/minidlna.conf (full doc here):

media_dir=P,/path/to/Pictures
media_dir=V,/path/to/Videos
media_dir=A,/path/to/Music
friendly_name=MediaServer

sudo systemctl enable minidlna

sudo systemctl start minidlna

Journalctl

If you used /var/log/syslog and dmesg before, it's like someone put it all in one tool including cats, greps and rotations, which is great.

I recommend this guide to start with: https://smarketshq.com/a-journalctl-mini-tutorial-8675a5daa072

The only thing that didn't work for me were the -u wildcards. E.g. journalctl -u postfix will only dump "postfix" and not "postfix/smtpd" lines, journalctl -u postfix* will just hang for some reason, journalctl -u postfix@-.service is the one you want. I found it out by typing "journalctl -u postfix" and hitting TAB.

Nano

sudo apt install nano

This is a semigraphical text editor that's much easier to navigate than vi.

Useful shortcuts:

• CTRL+K "cut" line
• CTRL+U "paste" line
• CTRL+O save
• CTRL+X exit

Screen

sudo apt install screen

A resident, multi-shell server that "survives" SSH sessions.

Just type "screen" - ENTER, you'll get a welcome screen with a few tips and then a shell. Unlike an SSH shell, this shell becomes a background process that runs regardless of whether you're logged in or not. You can detach, log off, turn your PC off, then turn it back on and log back in and pick up right where you left off.

"screen -r" resumes an existing "screen" session.

"screen -d" forcefully detaches any current connected or stale client.

Once you're in, you can press CTRL+A, release and then press a command key to interact with the "screen" server.

Most useful CTRL+A command keys:

• 'c' - create a new shell
• 'a' - switch to the last viewed shell
• '0'..'9' - switch to the corresponding shell number
• 'd' - detach (return to the spawning shell)