Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| selfhost:storage [2024/02/06 15:25] – created willy | selfhost:storage [Unknown date] (current) – removed - external edit (Unknown date) 127.0.0.1 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ===== Storage Setup ===== | ||
| - | |||
| - | The idea is to store your entire media collection on a redundant software RAID-1 array located on a different drive from the one where the Operating System is installed. In this way it will be easier to migrate to a new server in the future and your data will be safe from a disk failure. Now, there are different solutions you can choose from. You could go RAID-5, RAID-0+1, and many more combinations. RAID-1 has a few advantages for me which are: | ||
| - | * Good balance with wasted disk space (50% usage) | ||
| - | * Fast enough on read (reads will be balanced on both disks) | ||
| - | * Solid enough to survive one disk fail (provided you monitor the RAID status and replace failed disks) | ||
| - | |||
| - | If you feel like exploring different setups, go ahead. | ||
| - | |||
| - | As i said earlier, i have been using Linux software RAID implementation for more than one decade and i have never been let down. It's solid, it's simple, it works and it's efficient. If you choose to use a commercial external RAID solution, skip the RAID part ahead. I will assume you have two external drives called** /dev/sdb** and **/ | ||
| - | |||
| - | I will add some speed considerations: | ||
| - | |||
| - | Note: i will refer to //two// disks, but you can create more RAID arrays if you have //four// disks! | ||
| - | |||
| - | |||
| - | ===== Partitioning ===== | ||
| - | |||
| - | To create a software RAID, you need to first partition the two drives, for this job you can use the good old fdisk: | ||
| - | |||
| - | <code bash> | ||
| - | > su | ||
| - | > fdisk /dev/sdb | ||
| - | ... do the partitioning ... | ||
| - | > fdisk /dev/sdc | ||
| - | ... do the partitioning ... | ||
| - | </ | ||
| - | You will need to be root for fdisk to work. You should be root at this point, tough. The //su// command might be redundant. | ||
| - | |||
| - | Now, you need to initialize the disk (if it's a new disk), and in this case i suggest you initialize it as a GPT disk nowadays (fdisk command ' | ||
| - | |||
| - | (if you need to retain your data and you have only two disks, you can create the RAID only on one of the two, which will be deleted, mount with only one drive, copy the data over from the other disk, then format the other disk and hot-add it to the RAID-1. How to do this in details it's not difficult to figure out, but be careful not to lose your data in the process) | ||
| - | |||
| - | Using //fdisk//, create one partition on each drive to fill it, that will be called **/ | ||
| - | |||
| - | Save the changes and quit from fdisk, since the disks are not being used yet, you will not need to reboot the server. | ||
| - | |||
| - | |||
| - | ===== Creating the RAID array ===== | ||
| - | |||
| - | You need to create a new software RAID array out of **/ | ||
| - | |||
| - | <code bash> | ||
| - | > mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 | ||
| - | </ | ||
| - | |||
| - | since this is the first RAID array of the server (and probably the only one) it's called **/ | ||
| - | |||
| - | If you want to do the trick of copying existing data from one of the two disks, at this point you can create a RAID-1 array with only one drive by replacing the drive you do not want to add now with the work // | ||
| - | |||
| - | |||
| - | ===== Format and mount ===== | ||
| - | |||
| - | You need to format and mount your newly created RAID array. You need to choose which filesystem you want to use. A common choice on Linux, and probably the more straightforward, | ||
| - | |||
| - | I will be assuming your media collection is located under **/data** and so you should run the following commands: | ||
| - | |||
| - | <code bash> | ||
| - | > mkfs.ext4 /dev/md0 | ||
| - | > mkdir /data | ||
| - | > mount /dev/md0 /data | ||
| - | </ | ||
| - | |||
| - | **/data** will be the entry point for your media collection, you will be creating some subfolders structure soon enough. You will also store all the temporary files and executable files for the various software stack that you will be using so that everything but the actual Gentoo installation will be on the RAID array. This will speed up any hardware failure or reinstallation you might need in the future and will also ensure that your main Gentoo partition will not get filled up by the various tools and the downloaded data. | ||
| - | |||
| - | The newly formatted drive needs to be automatically mounted at every boot, so you need to add a line like this to **/ | ||
| - | |||
| - | < | ||
| - | / | ||
| - | </ | ||
| - | |||
| - | The //noatime// option will reduce USB traffic and wear-and-tear. | ||
| - | |||
| - | ===== Automate RAID at boot ===== | ||
| - | |||
| - | You also want to automate linux raid startup, so that upon reboot everything will still work just fine. To do so, the **mdraid** service needs to be started in the //boot// runlevel. Do NOT start it in the //default// runlevel or things will break badly after the first reboot. | ||
| - | |||
| - | <code bash> | ||
| - | > rc-update add mdraid boot | ||
| - | </ | ||
| - | |||
| - | The //mdadm// service is not required, unless you want monitoring of your RAID array (nicluding email reporting) which is a **TODO**. | ||
| - | |||
| - | You also want to ensure the **/ | ||
| - | |||
| - | < | ||
| - | ARRAY /dev/md0 UUID=1758bcfa: | ||
| - | </ | ||
| - | |||
| - | where the UUID can be read by the output of the command: | ||
| - | |||
| - | <code bash> | ||
| - | > mdadm -detail /dev/md0 | ||
| - | </ | ||
| - | |||
| - | |||
| - | ===== Prepare the disk for the media collection ===== | ||
| - | |||
| - | You need to create the individual collections entry points like: | ||
| - | |||
| - | <code bash> | ||
| - | > mkdir /data/Films | ||
| - | > mkdir /data/Tv | ||
| - | > mkdir /data/Music | ||
| - | > mkdir /data/Books | ||
| - | > mkdir / | ||
| - | > mkdir / | ||
| - | > mkdir / | ||
| - | </ | ||
| - | |||
| - | The last two ones will not contain actual media, but will be used to store the installation (or the cache folders) for the various software described in this page and the root folder for the reverse-proxy. | ||
| - | |||
| - | You will need do change the ownership of the folders, but you can do it after you have installed the *Arr' | ||
| - | |||
| - | <code bash> | ||
| - | > chgrp media -R /data/* | ||
| - | </ | ||