Differences

This shows you the differences between two versions of the page.

--- selfhost:fileserver [2024/01/18 14:08] – willy
+++ selfhost:fileserver [2025/03/19 15:09] (current) – willy
@@ Line 1: / Line 1: @@
-====== File Server ======
+====== H) File Server ======
+I will not discuss how to share your files on the home network using __legacy__ tools like [[https://it.wikipedia.org/wiki/Network_File_System|NFS]] or [[https://it.wikipedia.org/wiki/Samba_(software)|SAMBA]], there are plenty of tutorials online and, beside, it's kind out of the scope for self-hosting.
-Access to a **common area** (future: also user-specific **private** areas).
+I will focus on how to provide access via __web browser__ and via __WebDAV__, which is a web-based sharing protocol a bit like NFS or SAMBA, but aimed ad broader //in**ter**net// access, and not //in**tra**net// access.
-Access must be both from web page (HTTP/S) and from WebDAV.
+The idea is to create share areas where your users will be able to store files. It is possible to extend this idea also to user-specific areas where each user can put private stuff not visible by other users, but this require a little bit extra complexity and might be addressed in the future.
-[[https://filebrowser.org/|File Browser]] is used for web based access from browser.
+You will be using your SSO authentication, there will be no need to create new users anywhere, and it will of course be protected by the Reverse Proxy for external access.
-[[https://httpd.apache.org/|Apache]] is used as WebDAV server.
+In the past i used a more complex solution leveraging more tools. That obsolete solution has been moved, for reference, [[selfhost:obsolete:fileserver-legacy|here]].
-[[sailing:NGINX]] is used as reverse proxy to provide secure access to both WebDAV clients and browsers.
+===== Overall Architecture and Shares =====
+This solution leverages the use of one tool called AList (installation & configuration instructions [[services:alist|here]]). AList is a pretty neat open source tool which is capable to provide, at the same time, both a browser-based solution and a WebDAV solution to access your files.
-===== Permissions and Users =====
+AList itself also support SSO integration, opening the way to provide also a public sharing approach, if needed since the SSO should not be enabled at reverse-proxy level.
-All users need to be in the **users** group.
+You can also define as many shared folders as you like, and even connect to remote services from the same UI.
-The **common** share will be accessible by any user in the **users** group.
+I will assume that your shares are under **/data/shares**, but of course each share can be located anywhere you like. Let's also assume, as an example, that your share is called __/data/shares/common__ and is managed by the user __fileserver__ of the group __users__. The requirement for users and groups will be detailed later on.
-===== Shares Configuration =====
+Each share folder will have the following structure:
+  * /data/share/common/
+  * /data/share/other_share/
+  * /data/share/another_share/
-Files will be under **/home/common** for example. The shares will be configured in the **/etc/conf.d/shares** file:
+Your AList installation will provide WebDAV and browser access from one single port hwich need to be reverse-proxied.
-<file txt shares>
-SHARES="common:3002 other:3003"
-</file>
-where "common" and "other" is the name of the folder under **/home** and 3002/3003 is the port number (which will be needed for NGINX reverse proxy access via browser).
+I choose to assign a dedicated subdomain, **drive.mydomain.com**, as file server and organize the shares like this:
+  * **https://drive.mydomain.com**: will show a main login page to access all the shares
+  * **https://drive.mydomain.com/common**: direct browser access URL for __common__
+  * **https://drive.mydomain.com/webdav**: WebDAV access URL for all the shares
+  * **https://drive.mydomain.com/webdav/common**: WebDAV specific access URL for __common__
+  * **https://drive.mydomain.com/dav**: WebDAV access URL for all the shares
+  * **https://drive.mydomain.com/dav/common**: WebDAV specific access URL for __common__
+I think that /webdav is easier to remember than /dav, but AList by default shared WebDAV under /dav, NGINX will be used to map the /webdav path to /dav.
-===== Software Installation for Browser access =====
+You can add any more folders as separate shares as you like. Due to how WebDAV works, it is mandatory to separate the browser accessible URLs from the WebDAV ones, like i did above.
-[[https://filebrowser.org/|File Browser]] is a nice web-based file manager that you can use to access your file server via browser.
-I do not like the default installation method because it will install system-wide. I will show you how to install in a more customized way.
+=== Permissions and Users ===
-first you need to create a new user:
+(Note: you should run AList as the user **fileserver** and group **users**)
-<code bash>
- > useradd -d /data/daemons/filebrowser -m filebrowser -g users
-</code>
-the //filebrowser// user will have **users** as it's main group so that any files managed by it can be accessed and managed by users as well.
+I assume you have already created the user **fileserver** when installing AList.
-You will need to create the following folders architecture in your //filebrowser// home folder:
+You need to set the //umask// for the fileserver user to **0002** so that any new files created by it will be writable by the users:
-  * bin: where the FileBrowser binary will be located
-  * data/db: where the FileBrowser databases files will be stored
-  * data/logs: where the various log files will be created
-You need to set the //umask// for the user to **0002** so that any new files created by it will be writable by the users.
-Then, as //filebrowser// user, get the software package and decompress it. The default install approach is based on a auto executable web link ([[https://raw.githubusercontent.com/filebrowser/get/master/get.sh|here]]) which i do not recommend to use directly. Instead go to [[https://github.com/filebrowser/filebrowser/releases/|here]] and download the proper package for your architecture. Then:
 <code bash>
- > su - filebrowser
+mkdir /data/shares
- > echo "umask 0002" >> ~/.bashrc
+mkdir /data/shares/common
- > source ~/.bashrc
+chown fileserver:users /data/shares
- > mkdir bin data data/logs data/db
- > cd bin
- > tar xvf ../linux-amd64-filebrowser.tar.gz
 </code>
-Now, you will need to start a copy of FileBrowser for each share you want to have, and it must be owned by the user that want file permissions on that share. To achieve this, you will be using a special script called **fileserver.sh** which i will show you at the end, because it will contain also the WebDAV start stuff in it.
-===== Software Installation for WebDAV access =====
+===== Fileserver access via Browser =====
-While there are a few WebDAV servers like [[https://github.com/micromata/dave|Dave]], they seems to be either unmaintained or overly complicated. Also [[sailing:nginx|NGINX]] can be a WebDAV server, but it seems to be buggy and not supporting LOCK stuff, so i decided to go with Apache web server, which also has a long standing WebDAV implementation.
+Nothing extra needs to be done except install AList, and adding the new shares inside it's WEB configuration.
-The idea here is to run a dedicated copy of Apache as user //filebrowser// and group //users// so that it can access and manage the shared files. So first you need to emerge apache:
-<code bash>
- > emerge apache
-</code>
-WebDAV is enabled by default in Gentoo Apache ebuild.
-Running apache manually requires some effort, so, buckle up.
+===== Fileserver access via WebDAV =====
-First of all, Apache needs some folders to operate, so you need to create:
+__NOTE:__ using HTTP will cause a 301 redirect to HTTPS, and WebDAV clients will fail. So use HTTPS URL in webdav clients and not HTTP.
-  * /data/daemons/filebrowser/data/conf: to store the apache config file
-  * /data/daemons/filebrowser/data/roots: which will map as WebDAV root (you will see why)
-  * /data/daemons/filebrowser/data/locks: which will be used for WebDAV lock databases
-  * /data/daemons/filebrowser/data/pids: which will be used to store apache PID files
-<code bash>
+The only chnage you need to make is to add the following location to the NGINX configuration file you created during AList setup:
- > su - filebrowser
+<code>
- > mkdir /data/daemons/filebrowser/data/conf
+        location /webdav/ {
- > mkdir /data/daemons/filebrowser/data/root
+               proxy_pass http://127.0.0.1:5244/dav/;
-</code>
+               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+               proxy_set_header X-Forwarded-Proto $scheme;
-Then create the Apache config file for each share. You should create this config that will be used by each share **/data/daemons/filebrowser/data/conf/apache_global.conf**:
+               proxy_set_header Host $http_host;
-<file txt apache_global.conf>
+               proxy_set_header X-Real-IP $remote_addr;
-ServerRoot "/usr/lib64/apache2"
+               proxy_set_header Range $http_range;
-LoadModule actions_module modules/mod_actions.so
+               proxy_set_header If-Range $http_if_range;
-LoadModule alias_module modules/mod_alias.so
+               proxy_redirect off;
-LoadModule auth_basic_module modules/mod_auth_basic.so
+               client_max_body_size 20000m;
-LoadModule authn_anon_module modules/mod_authn_anon.so
-LoadModule authn_core_module modules/mod_authn_core.so
-LoadModule authn_dbm_module modules/mod_authn_dbm.so
-LoadModule authn_file_module modules/mod_authn_file.so
-LoadModule authz_core_module modules/mod_authz_core.so
-LoadModule authz_dbm_module modules/mod_authz_dbm.so
-LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
-LoadModule authz_host_module modules/mod_authz_host.so
-LoadModule authz_owner_module modules/mod_authz_owner.so
-LoadModule authz_user_module modules/mod_authz_user.so
-LoadModule autoindex_module modules/mod_autoindex.so
-<IfDefine CACHE>
-LoadModule cache_module modules/mod_cache.so
-</IfDefine>
-LoadModule dav_module modules/mod_dav.so
-LoadModule dav_fs_module modules/mod_dav_fs.so
-LoadModule dav_lock_module modules/mod_dav_lock.so
-LoadModule deflate_module modules/mod_deflate.so
-LoadModule dir_module modules/mod_dir.so
-LoadModule env_module modules/mod_env.so
-LoadModule expires_module modules/mod_expires.so
-LoadModule ext_filter_module modules/mod_ext_filter.so
-<IfDefine CACHE>
-LoadModule file_cache_module modules/mod_file_cache.so
-</IfDefine>
-LoadModule filter_module modules/mod_filter.so
-LoadModule headers_module modules/mod_headers.so
-<IfDefine HTTP2>
-LoadModule http2_module modules/mod_http2.so
-</IfDefine>
-LoadModule include_module modules/mod_include.so
-<IfDefine INFO>
-LoadModule info_module modules/mod_info.so
-</IfDefine>
-LoadModule log_config_module modules/mod_log_config.so
-TransferLog /deposito/archive/logs/common_transfer_log
-LoadModule logio_module modules/mod_logio.so
-LoadModule mime_module modules/mod_mime.so
-LoadModule mime_magic_module modules/mod_mime_magic.so
-LoadModule negotiation_module modules/mod_negotiation.so
-LoadModule rewrite_module modules/mod_rewrite.so
-LoadModule setenvif_module modules/mod_setenvif.so
-<IfDefine STATUS>
-LoadModule status_module modules/mod_status.so
-</IfDefine>
-LoadModule unique_id_module modules/mod_unique_id.so
-LoadModule unixd_module modules/mod_unixd.so
-<IfDefine USERDIR>
-LoadModule userdir_module modules/mod_userdir.so
-</IfDefine>
-LoadModule usertrack_module modules/mod_usertrack.so
-LoadModule vhost_alias_module modules/mod_vhost_alias.so
-Include /etc/apache2/modules.d/*.conf
-</file>
-Then you can create one config file for each share. This is the file for the common share **/data/daemons/filebrowser/data/conf/common.conf**:
-<file txt common.conf>
-Include /data/daemons/filebrowser/data/conf/apache_global.conf
-User filebrowser
-Group users
-DavLockDB "/data/daemons/filebrowser/data/locks/common"
-PidFile /data/daemons/filebrowser/data/pids/common.pid
-ErrorLog /data/daemons/filebrowser/data/logs/common_error_log
-TransferLog /data/daemons/filebrowser/data/logs/common_transfer_log
-CustomLog /data/daemons/filebrowser/data/logs/common_access_log common
-DocumentRoot /data/daemons/filebrowser/data/roots
-ServerName 127.0.0.1
-Listen 127.0.0.1:10001
-<Directory /data/daemons/filebrowser/data/roots>
-    DAV On
-    AllowOverride All
-    Options -Indexes +FollowSymlinks -ExecCGI -Includes
-    Require all granted
-</Directory>
-SetEnv redirect-carefully
-# vim: ts=4 filetype=apache
-</file>
-Please note the Listen directive: you want apache to be bound to 127.0.0. only and note the port too, this port will be needed for the reverse proxy. Each share will need it's own port.
-Now, the fun part is that you want to protect this behind the NGINX reverse proxy and it seems that WebDAV does **not** play well with URL redirection and similar funny things. In other words, the base url you will be using on the reverse proxy must match the url in the Apache. You cannot use rewrite directives or Alias stuff.
-Since you will be exposing the browser-based access as **https://your_server/archive/common** and the WebDAV access as **https://your_server/webdav/common** it means that we need to sym-link your **/home/common folder to /data/daemons/filebrowser/data/roots/webdav/common** for it to work:
-<code bash>
- > su - filebrowser
- > cd data/root
- > mkdir webdav
- > cd webdav
- > ln -s /home/common common
-</code>
-Now it's time to finish up.
-===== Reverse Proxy =====
-You want to integrate all this into the SSL enabled reverse proxy, which is also using PAM authentication.
-Now, reverse proxy is simple, but this into **/etc/nginx/folders/filebrowser.conf**:
-<file txt filebrowser.conf>
-  # Browser based access here
-  location /archive/common/ {
-        client_max_body_size 512M;
-        proxy_pass http://127.0.0.1:3002;
-        proxy_http_version 1.1;
-        proxy_set_header Connection $http_connection;
-        proxy_set_header Connection 'upgrade';
-        proxy_cache_bypass $http_upgrade;
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-    }
-    # WebDAV access
-    location /webdav/common {
-        # https://mailman.nginx.org/pipermail/nginx/2007-January/000504.html - fix Destination: header
-        # https://trac.nginx.org/nginx/ticket/348 - bug, workaround with named capture
-        set $dest $http_destination;
-        if ($http_destination ~ "^https://(?<myvar>(.+))") {
-                set $dest http://$myvar;
         }
-        proxy_pass http://127.0.0.1:10001;
-        proxy_redirect off;
-        proxy_buffering off;
-        gzip off;
-        proxy_pass_request_headers on;
-        proxy_set_header Destination       $dest;
-        proxy_set_header      Host $host;
-        proxy_set_header      X-Real-IP $remote_addr;
-        proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
-}
-</file>
-and put this file include inside the usual **/etc/nginx*/folders/main.conf**, and restart nginx.
-<file bash filebrowser.sh>
-source /etc/conf.d/shares
-BASE_PATH=/deposito/daemons/filebrowser/data
-for i in $SHARES
-do
-        SHARE=$(echo $i | cut -d: -f1)
-        PORT=$(echo $i | cut -d: -f2)
-        OWNER=filebrowser
-        echo Starting FileBrowser for $OWNER on share $SHARE
-        su - $OWNER -c "/deposito/daemons/filebrowser/bin/filebrowser config set --auth.method=noauth -d $BASE_PATH/db/filebrowser_$SHARE.db >/dev/null"
-        su - $OWNER -c "/deposito/daemons/filebrowser/bin/filebrowser -r /deposito/$SHARE -p $PORT -b /archive/$SHARE -d $BASE_PATH/db/filebrowser_$SHARE.db -l $BASE_PATH/logs/filebrowser_$SHARE.log 2> $BASE_PATH/logs/filebrowser_${SHARE}_run.log"&
-        echo Starting WebDAV backend for $OWNER on share $SHARE
-        su - $OWNER -c "apache2 -f /data/daemons/filebrowser/data/conf/$SHARE.conf"
-done
-</file>
-And the usual autostart stuff:
-<file bash 40-filebrowser.start>
-#!/bin/bash
-/data/daemons/filebrowser/fileserver.sh
-</file>
-Make both files executable.
-===== Background =====
-From users point of view, the common area will be managed by user **filebrowser** which is designed to run as group **users** with an //umask// 550 so that any files uploaded via web browser will be accessible to the normal users.
-Of course, each user will need to be part of the **users** group as well.
-You will need a common "archive" folder under **/home/common** to store some needed stuff.
-This folder will need to contain:
-  * **common** subfolder, where the common area files will be stored (created in the [[sailing:filebrowser]] instructions)
-  * **temp/uploads** subfolder, required by WebDAV to upload files
-  * **temp/pids** subfolder, to store NGINX pids
-  * **temp/tmp** subfolder, to store NGINX run files
-  * **logs** subfolder, to store NGINX log files
-  * **conf** subfolder, where you will store custom NGINX config files for the private areas (and common area too)
-Create the folders:
-<code bash>
- > mkdir /data/archive
- > mkdir /data/archive/logs
- > mkdir /data/archive/common
- > mkdir /data/archive/temp
- > mkdir /data/archive/temp/pids
- > mkdir /data/archive/temp/tmp
- > mkdir /data/archive/conf
- > chown filebrowser:users -R /data/archive
 </code>
-===== NGINX WebDAV approach =====
+which will remap /webdav to /dav
-No need to use third party WebDAV server since NGINX has a pretty solid implementation of it already. Follow the [[sailing:nginx]] instructions to set NGINX up with WebDAV and PAM auth support.
-Now, there is a nasty catch here which stems from using NGINX as WebDAV server... You need to run NGINX as //filebrowser// user to ensure that the access trough WebDAV will not incur in access permissions errors. Running NGINX as standalone user requires the creation of a specific nginx.conf with some specifics in it.
-For consistency, this NGINX config file will be **/data/archive/conf/nginx_common.conf**:
-<file txt /data/archive/conf/nginx_common.conf>
-worker_processes 1;
-pid /data/archive/temp/pids/nginx_common.pid;
-error_log /data/archive/logs/common_error_log info;
-events {
-        worker_connections 100;
-        use epoll;
-}
-http {
-        include /etc/nginx/mime.types;
-        default_type application/octet-stream;
-        # These folder MUST be redirected to avoid usage of system wide ones:
-        client_body_temp_path  /data/archive/temp/tmp;
-        proxy_temp_path  /data/archive/temp/tmp;
-        fastcgi_temp_path  /data/archive/temp/tmp;
-        uwsgi_temp_path  /data/archive/temp/tmp;
-        scgi_temp_path  /data/archive/temp/tmp;
-        disable_symlinks off;
-        keepalive_timeout 75 20;
-        server {
-                server_name 127.0.0.1;
-                access_log /data/archive/logs/common_access_log;
-                location / {
-                        root /data/archive/common/;
-                        dav_methods PUT DELETE MKCOL COPY MOVE;
-                        dav_ext_methods PROPFIND OPTIONS;
-                        dav_access user:rw group:rw all:r;
-                        client_max_body_size 0;
-                        create_full_put_path on;
-                        client_body_temp_path /data/archive/uploads;
-                }
-                listen 10000;
-        }
-}
-</file>
-This NGINX server will listen on 127.0.0.1:10000, and you will need to setup a reverse proxy from the main NGINX, by creating the following config file **/etc/nginx/folders/webdav.conf**:
-<file txt webdav.conf>
-location ~ ^/webdav/common {
-        rewrite /webdav/common/(.*) /$1 break;
-        proxy_pass http://127.0.0.1:10000;
-}
-</file>
-and including it into the main NGINX server.
-Now, edit the **/data/daemons/filebrowser/filebrowser.sh** file and add the following line:
-<code>
-nginx -c /deposito/archive/conf/nginx_common.conf -e /deposito/archive/logs/common_error_log
-</code>
-like this:
-<file bash filebrowser.sh>
-#!/bin/bash
-cd /data/daemons/filebrowser &&
-nginx -c /data/archive/conf/nginx_common.conf -e /data/archive/logs/common_error_log
-./filebrowser -r /data/archive/common -p 3002 -b /archive/common 2>&1 > filebrowser.log
-</file>
-and restart filebrwoser and the main NGINX.
-At this point, your common area will be ready and working both on WebDAV and directly via web browser.
-To access via browser:
-to access via WebDAV clients:

Willy's Wiki

User Tools

Differences