====== Photos Management Server ====== Photos (and videos!) are something very important in our everyday's life. They collect our memories and can bring back the emotions that we all experiences just by reviewing the pictures and videos taken back in time. Photography has greatly changed over the years. For example i have thousands of photos in the last 20+ years with many different cameras and technologies: film SLRs, DSLRs, phones, disppsable cameras... Also the digital management tools have changed greatly. I even have 40+ years old family photos that i have scanned and stored. Today the good old paper albums are just collecting dust in the bookshelves, and tomorrow those will be a faded memory of the past. Today photos are shared over instant messaging apps and given for free to big corps like Google and Apple. The challenge is to make the family album fun again, keep it private and make it last our grandchildren (maybe). ====== The concepts ====== the goal is to have the family photos neatly organized, sorted and easily accessible by family members, even (specially) the non tech-savvy ones. There are three steps to the process: * Collecting pictures and videos from the devices (cameras, phones, scans...) * Organizing them into albums and collections, adding tags if/when possible * Display them all for easy fruition, possibly with Machine Learning for face detection and such ===== Long term support & availability ===== Tehcnology changes, fast. Applications are created new and dies out. The [[https://en.wikipedia.org/wiki/Bus_factor|bus factor]] is vividly true also for Open Source projects. I have stored photos from 40 years ago, and literally thousands of pictures from 20+ years ago. To be usable they **must** be organized, tagged and properly labelled. Such tagging and organization **must** survive each single app or tool you choose. Using a database is **not** acceptable, since as much as logical it might be organized, you cannot extract and parse the tags&metadata easily after the app that uses it get discontinued, or you die (yes, i am in that time of life where legacy start to be a tought). As an example, i have an extensive database of public photos i did some years ago. The tool itself does not work anymore, but i still retain the photos, whose names have been mangled for "storage reasons" by the tool, and the entire SQL dump of the database. I //could// digets the db and create all the needed queries to sort out again the photos in albums and associate the tags, but i never did it because it is so much pain. Lesson learnt. The only viable option (CHANGE MY MIND) is having the photos organized in folders, where each folder is an album, and the folders themselves named in a clear, obvious, album title way. The tags and metadata should be either embedded in the photo files or stored in sidecar files along the photos themselves. This is non-negotiable for me. If i didn't follow this rule since i started digitalizing my photos, today i would have nothing still readable and organized. and what about in 10 more years? ===== Tools & software ===== Let's be clear: there no tool that let you do that all-in-one, and i have tested almost all of them. First of all, i discarded all **local** tools, as i want this done on the server and not on my local laptop. Ideally, every family member should be able to to it's part without //my// intervention. (back in the past i used the great [[https://www.digikam.org/|Digikam]] which still rocks, but it's local-only). On the **web** tools the landscape is a little bit better than for the [[services:notes|Note Taking]] category, but not that much. There are quite a few tools indeed, but each one has it's pros and cons, and none fits the bill perfectly. I have tried (for each one, details in the associated page): * [[services:immich|Immich]] is the most promising, but doesn't support folder-albums (without a complex and not intuitive usage pattern with templates), and overall i don't like that it cannot write back the ML generated data to the photos or sidecar files. * [[services:damselfly|Damselfly]] is a Digital Asset Management tool. Very impressive for a single developer, but a bit not user-friendly and limited in what it can do * [[services:homegallery|HomeGallery]] is fantastic gallery with an innovative and intuitive browsing approach. Unfortunately does not support folder-albums. * [[services:librephotos|LibrePhotos]] is nice, but extremely slow in indexing and honestly, it seems a bit abandoned. * [[services:photofield|Photofield]] has some impressive view tricks, but it's way too simple. * [[services:pigallery2|PiGallery2]] is a great gallery, supports tags and such, but it's not able (by design) to manage/move photos * [[services:piwigo|Piwigo]] in theory is powerful and can do it all, but in relaity feels old, dated and ugly. It's been around 20+ years, and in this case this is a downpoint. * [[services:photoview|PhotoView]] is a photo album designed for photographers by a photographer, and seems by far, at this time, the best option. It support sharing albums to the public and face recognition, in addtion to folder views and such. There are many more, which i didn't try because they where either **dead**, **dying** or non really open-source (as i stated many times, i try to steer from projects that are open-source only to generate a revenue). ====== The Architecture & Workflow ====== All the photos will be stored on the server RAID array (remember: this is **not** a backup, don't forget to [[selfhost:backup|Backup]] your photos!) on a three levels subfolder main structure located under **/data/Photos**: * /data/Photos/Staging: here your new photos will be uploaded * /data/Photods/Gallery: photos here will be visible in the gallery These folder could be netowork-shared inside your home network in case you want to manage your photos via a local tool like Digikam, plus they need to be accessible via Filebrowser so that they can be managed via web too. From a web point of view i suggest you create a new subdomain **photos.mydomain.com** for the gallery itself, while the organization URL could be placed under **drive.mydomain.com/photos** to leverage the work already done on the [[services:fileserver|File Server]] side of things. The **Staging** folder will need to have one subfolder for each user, to keep uploaded photos not overlapping. The **Gallery** folder can be organized in any way you want ot organize your photo collection. The workflow will be: * Take photos and upload them to the server (automatically, when possible, using Syncthing) * Organize the photos on the server, via web interface (FileBrowser), into albums/folders * Have the folder published to the gallery app (PhotoView) that ease browsing, tagging and such. === Step 0: preparing users and folders === For simple but secure management, you will need a dedicated user called **photos**, member of the **photos** group. For added accessiblity, you could want to add all or some of your users to the same group, so that you can mount the photo storage on your computers and use local tools like digikam too: groupadd photos useradd -d /data/daemons/photos -m -g photos photos Now you need to create the storage folders: mkdir /data/Photos chown photos:photos /data/Photos su - photos echo "umask 0003" >> ~/.bashrc source ~/.bashrc mkdir /data/Photos/Staging mkdir /data/Photos/Gallery The umask setting is to ensure photos are accessible by other photos users, YMMV, but i suggest you give world access to folders (the 3 in the above umask) so that containers can access your photos. === Step 1: configure how to load stuff in the Staging Area === At first i tried to use //Immich// for this job. The storage templates are somehow mandatory to this end, and it might just work for you. But i ended up with a more practical and effective approach since i am already using [[gentoo:syncthing|SyncThing]] on all our phones. Feel free to use either system, or even another one. I will describe here the Syncthing approach. The advantage of Syncthing is that you can use it not only on mobile phones/tablets but also on laptops and different devices. In order to do so, you need to spin a second copy of Syncthing on a different port just for the photo users, since your main Syncthing copy is running with a different user. There are other ways to do this, but i think they are less secure. The [[gentoo:syncthing|SyncThing]] page will tell you how to run Syncthing in general, but here is a quick summary of what you need to do: * Copy **/etc/conf.d/syncthing** to **/etc/conf.d/syncthing.photos** * Symlink **/etc/init.d/syncthing** to **/etc/init.d/syncthing.photos** * Edit **/etc/conf.d/syncthing.photos** (see below) * Run //once// "syncthing" as user //photos// to initialize configuration files, then stop it * Start the service as usual The configuration file should look something like: SYNCTHING_USER="photos" SYNCTHING_GROUP="photos" SYNCTHING_HOMEDIR="/data/daemons/photos/.config/syncthing" SYNCTHING_LOGFILE="/var/log/syncthing/photos.log" SYNCTHING_GUI_ADDRESS="http://127.0.0.1:8385" (note the //8385// port that must be different from the default //8384// not to clash with the main syncthing installation) To add&start the service: rc-update add syncthing.photos default /etc/init.d/syncthing.photos start As usual, install Syncthing on your phone too and link them together (see Syncthing documentation) and add one synced folder from your /DCIM/Camera in Android to something like /data/Photos/Staging/Myself-sync. Remeber you want to separate each user in the staging area to avoid mess. If you want to use Immich instead (or in addition to), check [[services:immich|this]] page on how to install it, just make Immich upload photos inside your **/data/Photos/Staging/**. === Step 2: setup how to organize the Staging Area files (and move to the Gallery) === It is not a good idea to directly upload to the gallery because out mobile devices usually take lots of crap photos that you do not want to store. That's why i use a staging folder. After the photos have been uploaded to the staging area, you will need to access it, sort the photos and move to the Gallery area. You don't need to do it often. I mostly do that after an event or an occasion where i take photos and want to organize them to place in the family photo memories. You can always do so by mounting the photo folders on a local PC and use some photo manangement tools (like Digikam), but the highlight here is that you can as well do that over the web, even while on vacation or from your phone/tablet. To do so, i created a dedicated shared folder accessible via browser (see [[services:fileserver|File Server]] for more details) which uses Filebrowser, a neat web-browser. So i assume you create a share called **photos**, and this is the relevant config script under **/etc/conf.d/iflebrowser.photos**: BASE_URL="/photos" DATABASE="/data/daemons/photos/filebrowser.db" DESCRIPTION="Photo web archive management" FOLDER="/data/Photos" USER="photos" GROUP="users" PORT=3006 Create the link and start / autostart the service: ln -s /etc/init.d/filebrowser /etc/init.d/filebrowser.photos rc-update add filebrowser.photos default /etc/init.d/filebrowser.photos start Don't forget to add the share to your NGINX reverse proxy as described in the dedicated file server page. === Step 3: setup the gallery === As i stated at the beginning there are many gallery tools out there. My pick is, at this time: * PhotoView (install instructions [[services:photoview|here]]) is fast and natively show your album folders, can share photos and album publicly and much more. Go ahead, install it! === Step 4: backup === Backing up your photos is a **must**. Nothing so far is a //backup//, so check [[selfhost:backup|Backup]] and setup the backup of your **gallery** area. You sohuld probably not backup the **staging** area as it might change too frequently. YMMV, decide for yourself.