Photos Management Server
Photos (and videos!) are something very important in our everyday's life. They collect our memories and can bring back the emotions that we all experiences just by reviewing the pictures and videos taken back in time.
Photography has greatly changed over the years. For example i have thousands of photos in the last 20+ years with many different cameras and technologies: film SLRs, DSLRs, phones, disppsable cameras… Also the digital management tools have changed greatly. I even have 40+ years old family photos that i have scanned and stored.
Today the good old paper albums are just collecting dust in the bookshelves, and tomorrow those will be a faded memory of the past. Today photos are shared over instant messaging apps and given for free to big corps like Google and Apple.
The challenge is to make the family album fun again, keep it private and make it last our grandchildren (maybe).
The concepts
the goal is to have the family photos neatly organized, sorted and easily accessible by family members, even (specially) the non tech-savvy ones.
There are three steps to the process:
- Collecting pictures and videos from the devices (cameras, phones, scans…)
- Organizing them into albums and collections, adding tags if/when possible
- Display them all for easy fruition, possibly with Machine Learning for face detection and such
Long term support & availability
Tehcnology changes, fast. Applications are created new and dies out. The bus factor is vividly true also for Open Source projects. I have stored photos from 40 years ago, and literally thousands of pictures from 20+ years ago. To be usable they must be organized, tagged and properly labelled. Such tagging and organization must survive each single app or tool you choose.
Using a database is not acceptable, since as much as logical it might be organized, you cannot extract and parse the tags&metadata easily after the app that uses it get discontinued, or you die (yes, i am in that time of life where legacy start to be a tought). As an example, i have an extensive database of public photos i did some years ago. The tool itself does not work anymore, but i still retain the photos, whose names have been mangled for “storage reasons” by the tool, and the entire SQL dump of the database. I could digets the db and create all the needed queries to sort out again the photos in albums and associate the tags, but i never did it because it is so much pain. Lesson learnt.
The only viable option (CHANGE MY MIND) is having the photos organized in folders, where each folder is an album, and the folders themselves named in a clear, obvious, album title way. The tags and metadata should be either embedded in the photo files or stored in sidecar files along the photos themselves.
This is non-negotiable for me. If i didn't follow this rule since i started digitalizing my photos, today i would have nothing still readable and organized. and what about in 10 more years?
Tools & software
Let's be clear: there no tool that let you do that all-in-one, and i have tested almost all of them.
First of all, i discarded all local tools, as i want this done on the server and not on my local laptop. Ideally, every family member should be able to to it's part without my intervention. (back in the past i used the great Digikam which still rocks, but it's local-only).
On the web tools the landscape is a little bit better than for the Note Taking category, but not that much. There are quite a few tools indeed, but each one has it's pros and cons, and none fits the bill perfectly.
I have tried (for each one, details in the associated page):
- Immich is the most promising, but doesn't support folder-albums (without a complex and not intuitive usage pattern with templates), and overall i don't like that it cannot write back the ML generated data to the photos or sidecar files.
- Damselfly is a Digital Asset Management tool. Very impressive for a single developer, but a bit not user-friendly and limited in what it can do
- HomeGallery is fantastic gallery with an innovative and intuitive browsing approach. Unfortunately does not support folder-albums.
- LibrePhotos is nice, but extremely slow in indexing and honestly, it seems a bit abandoned.
- Photofield has some impressive view tricks, but it's way too simple.
- PiGallery2 is a great gallery, supports tags and such, but it's not able (by design) to manage/move photos
- Piwigo in theory is powerful and can do it all, but in relaity feels old, dated and ugly. It's been around 20+ years, and in this case this is a downpoint.
- PhotoView is a photo album designed for photographers by a photographer, and seems by far, at this time, the best option. It support sharing albums to the public and face recognition, in addtion to folder views and such.
There are many more, which i didn't try because they where either dead, dying or non really open-source (as i stated many times, i try to steer from projects that are open-source only to generate a revenue).
The Architecture & Workflow
All the photos will be stored on the server RAID array (remember: this is not a backup, don't forget to Backup your photos!) on a three levels subfolder main structure located under /data/Photos:
- /data/Photos/Staging: here your new photos will be uploaded
- /data/Photods/Gallery: photos here will be visible in the gallery
These folder could be netowork-shared inside your home network in case you want to manage your photos via a local tool like Digikam, plus they need to be accessible via Filebrowser so that they can be managed via web too.
From a web point of view i suggest you create a new subdomain photos.mydomain.com for the gallery itself, while the organization URL could be placed under drive.mydomain.com/photos to leverage the work already done on the File Server side of things.
The Staging folder will need to have one subfolder for each user, to keep uploaded photos not overlapping. The Gallery folder can be organized in any way you want ot organize your photo collection.
The workflow will be:
- Take photos and upload them to the server (automatically, when possible, using Syncthing)
- Organize the photos on the server, via web interface (FileBrowser), into albums/folders
- Have the folder published to the gallery app (PhotoView) that ease browsing, tagging and such.
Step 0: preparing users and folders
For simple but secure management, you will need a dedicated user called photos, member of the photos group. For added accessiblity, you could want to add all or some of your users to the same group, so that you can mount the photo storage on your computers and use local tools like digikam too:
groupadd photos useradd -d /data/daemons/photos -m -g photos photos
Now you need to create the storage folders:
mkdir /data/Photos chown photos:photos /data/Photos su - photos echo "umask 0002" >> ~/.bashrc source ~/.bashrc mkdir /data/Photos/Staging mkdir /data/Photos/Gallery
(the umask setting is to ensure photos are accessible by other photos users, YMMV)
Step 1: configure how to load stuff in the Staging Area
At first i tried to use Immich for this job. The storage templates are somehow mandatory to this end, and it might just work for you. But i ended up with a more practical and effective approach since i am already using SyncThing on all our phones. Feel free to use either system, or even another one. I will describe here the Syncthing approach.
The advantage of Syncthing is that you can use it not only on mobile phones/tablets but also on laptops and different devices.
In order to do so, you need to spin a second copy of Syncthing on a different port just for the photo users, since your main Syncthing copy is running with a different user. There are other ways to do this, but i think they are less secure.
The SyncThing page will tell you how to run Syncthing in general, but here is a quick summary of what you need to do:
- Copy /etc/conf.d/syncthing to /etc/conf.d/syncthing.photos
- Symlink /etc/init.d/syncthing to /etc/init.d/syncthing.photos
- Edit /etc/conf.d/syncthing.photos (see below)
- Run once “syncthing” as user photos to initialize configuration files, then stop it
- Start the service as usual
The configuration file should look something like:
- syncthing.photos
SYNCTHING_USER="photos" SYNCTHING_GROUP="photos" SYNCTHING_HOMEDIR="/data/daemons/photos/.config/syncthing" SYNCTHING_LOGFILE="/var/log/syncthing/photos.log" SYNCTHING_GUI_ADDRESS="http://127.0.0.1:8385"
(note the 8385 port that must be different from the default 8384 not to clash with the main syncthing installation)
To add&start the service:
rc-update add syncthing.photos default /etc/init.d/syncthing.photos start
As usual, install Syncthing on your phone too and link them together (see Syncthing documentation) and add one synced folder from your /DCIM/Camera in Android to something like /data/Photos/Staging/Myself-sync. Remeber you want to separate each user in the staging area to avoid mess.
If you want to use Immich instead (or in addition to), check this page on how to install it, just make Immich upload photos inside your /data/Photos/Staging/<username>.
Step 2: setup how to organize the Staging Area files (and move to the Gallery)
It is not a good idea to directly upload to the gallery because out mobile devices usually take lots of crap photos that you do not want to store. That's why i use a staging folder. After the photos have been uploaded to the staging area, you will need to access it, sort the photos and move to the Gallery area. You don't need to do it often. I mostly do that after an event or an occasion where i take photos and want to organize them to place in the family photo memories.
You can always do so by mounting the photo folders on a local PC and use some photo manangement tools (like Digikam), but the highlight here is that you can as well do that over the web, even while on vacation or from your phone/tablet.
To do so, i created a dedicated shared folder accessible via browser (see File Server for more details) which uses Filebrowser, a neat web-browser.
So i assume you create a share called photos, and this is the relevant config script under /etc/conf.d/iflebrowser.photos:
- filebrowser.photos
BASE_URL="/photos" DATABASE="/data/daemons/photos/filebrowser.db" DESCRIPTION="Photo web archive management" FOLDER="/data/Photos" USER="photos" GROUP="users" PORT=3006
Create the link and start / autostart the service:
ln -s /etc/init.d/filebrowser /etc/init.d/filebrowser.photos rc-update add filebrowser.photos default /etc/init.d/filebrowser.photos start
Don't forget to add the share to your NGINX reverse proxy as described in the dedicated file server page.
Step 3: setup the gallery
As i stated at the beginning there are many gallery tools out there. My pick is, at this time:
- PhotoView (install instructions here) is fast and natively show your album folders, can share photos and album publicly and much more.
Go ahead, install it!
Step 4: backup
Backing up your photos is a must. Nothing so far is a backup, so check Backup and setup the backup of your gallery area. You sohuld probably not backup the staging area as it might change too frequently. YMMV, decide for yourself.