Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| selfhost:fileserver [2024/01/18 14:20] – [Wrapup and Autostart] willy | selfhost:fileserver [2025/03/13 09:29] (current) – created - external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== File Server ====== | + | ====== |
| + | I will not discuss how to share your files on the home network using __legacy__ tools like [[https:// | ||
| - | Access | + | I will focus on how to provide access via __web browser__ and via __WebDAV__, which is a web-based sharing protocol a bit like NFS or SAMBA, but aimed ad broader //in**ter**net// access, and not //in**tra**net// access. |
| - | Access must be both from web page (HTTP/ | + | The idea is to create share areas where your users will be able to store files. It is possible to extend this idea also to user-specific areas where each user can put private stuff not visible by other users, but this require a little bit extra complexity |
| - | [[https:// | + | You will be using your SSO authentication, |
| - | [[https:// | ||
| - | [[sailing: | + | ===== Overall Architecture |
| + | This solution leverages different tools: | ||
| + | * [[services: | ||
| + | * [[services: | ||
| + | * [[https:// | ||
| - | ===== Permissions and Users ===== | + | Note: choosing between FileBrowser or Cloud Commander is a matter of preference. I use both, for different kind of shares. |
| - | All users need to be in the **users** group. | + | The NGINX reverse proxy will integrate with your preferred [[selfhost: |
| - | The **common** share will be accessible | + | I will assume that your shares are under **/shares**, but of course each share can be located anywhere you like. Let's also assume, as an example, that your share is called __/ |
| - | ===== Shares Configuration ===== | + | Each share folder will have the following structure: |
| + | * / | ||
| + | * / | ||
| + | * / | ||
| + | * / | ||
| - | Files will be under **/ | + | This structure is provided as an example to follow, of course you can move the individual folders where you prefer. The only caveat is that, for security reasons, the **db** and **webdav** folder should **not** be inside |
| - | <file txt shares> | + | |
| - | SHARES=" | + | |
| - | </ | + | |
| - | where "common" and " | + | You will also need to assign two ports for each share, as an example for our //common// share: |
| + | | ||
| + | * 10001: | ||
| + | Any other share can start from these port numbers and go up in numbering. | ||
| - | ===== Software Installation | + | I choose to assign a dedicated subdomain, **drive.mydomain.com**, |
| + | * **https:// | ||
| + | * **https:// | ||
| + | * **https:// | ||
| - | [[https:// | + | You can add any more folders as separate shares as you like. Due to how WebDAV works, it is mandatory |
| - | I do not like the default installation method because it will install system-wide. I will show you how to install in a more customized way. | ||
| - | first you need to create a new user: | + | === Permissions and Users === |
| - | <code bash> | + | |
| - | > useradd -d / | + | |
| - | </ | + | |
| - | the // | + | (Note: you should set both FileBrowser and Cloud Commander with user **fileserver**) |
| - | You will need to create the following folders architecture in your // | + | Each share will be accessible by different users, so this needs to be planned a bit. For user-specific shares, not much needs to be done except run FileBrowser/ |
| - | * bin: where the FileBrowser binary will be located | + | |
| - | * data/db: where the FileBrowser | + | |
| - | * data/logs: where the various log files will be created | + | |
| - | You need to set the //umask// for the user to **0002** so that any new files created by it will be writable by the users. | + | For common shares instead, it's important |
| - | Then, as // | + | You need to assign that folder |
| <code bash> | <code bash> | ||
| - | > su - filebrowser | + | useradd |
| - | > echo "umask 0002" >> ~/.bashrc | + | |
| - | > source ~/.bashrc | + | |
| - | > mkdir bin data data/logs data/db | + | |
| - | > cd bin | + | |
| - | > tar xvf ../linux-amd64-filebrowser.tar.gz | + | |
| </ | </ | ||
| - | Now, you will need to start a copy of FileBrowser | + | You need to set the // |
| + | <code bash> | ||
| + | su - fileserver | ||
| + | echo "umask 0002" >> ~/.bashrc | ||
| + | source ~/.bashrc | ||
| + | mkdir db | ||
| + | mkdir webdav | ||
| + | mkdir data | ||
| + | </ | ||
| - | ===== Software Installation for WebDAV access ===== | ||
| - | While there are a few WebDAV servers like [[https:// | + | ===== Fileserver access via Browser ===== |
| - | The idea here is to run a dedicated copy of Apache as user //filebrowser// and group //users// so that it can access and manage the shared files. So first you need to emerge apache: | + | Both [[services: |
| + | |||
| + | You can find installation instruction for both tools in the links above. Install both or the one you prefer, i will assume you have installed your pick on your system already by following my guides above. | ||
| + | |||
| + | You will need to run **one** instance of the tool you choose for //each share//, so you will to allocate one specific port for each share. I will describe how to run it for the **common** share, so the tool will run as the **fileserver** user that you created above. | ||
| + | |||
| + | If you choose FileBrowser: | ||
| + | So, create the specific **/ | ||
| + | <file - filebrowser.common> | ||
| + | BASE_URL="/ | ||
| + | DATABASE="/ | ||
| + | DESCRIPTION=" | ||
| + | FOLDER="/ | ||
| + | GROUP=" | ||
| + | PORT=3002 | ||
| + | USER=" | ||
| + | </ | ||
| + | |||
| + | If you choose Cloud Commander: | ||
| + | So, create the specific **/ | ||
| + | <file - cloudcmd.common> | ||
| + | BASE_URL="/ | ||
| + | DESCRIPTION=" | ||
| + | FOLDER="/ | ||
| + | GROUP=" | ||
| + | PORT=3002 | ||
| + | USER=" | ||
| + | </ | ||
| + | |||
| + | Create the **init.d** symlink too, and start it. Of course, choose a free port (3002). | ||
| + | |||
| + | |||
| + | ===== Fileserver access via WebDAV ===== | ||
| + | |||
| + | __NOTE:__ using HTTP will cause a 301 redirect to HTTPS, and WebDAV clients will fail. So use HTTPS URL in webdav clients and not HTTP. | ||
| + | |||
| + | While there are a few WebDAV servers like [[https:// | ||
| + | |||
| + | The idea here is to run a dedicated copy of Apache as user //fileserver// and group //users// so that it can access and manage the shared files. So first you need to emerge apache: | ||
| <code bash> | <code bash> | ||
| - | > | + | emerge apache |
| </ | </ | ||
| - | WebDAV is enabled by default in Gentoo Apache ebuild. | + | WebDAV is enabled by default in Gentoo Apache ebuild, so there is no need to fix USE flags. |
| - | Running apache manually requires some effort, so, buckle up. | + | You will **not** be running Apache as system service, because that will mess with our user permission approach. I have prepared the following init script that manages to start separated Apache copies for each of your shares. Do drop the following file to **/etc/init.d/ |
| + | <file - webdav> | ||
| + | # | ||
| + | # Copyright 2024 Willy Garidol | ||
| + | # Distributed under the terms of the GNU General Public License v3 | ||
| - | First of all, Apache needs some folders to operate, so you need to create: | + | depend() { |
| - | * / | + | need localmount net |
| - | * / | + | } |
| - | * / | + | |
| - | * / | + | |
| + | # Name of the share | ||
| + | WD_SHARE_NAME=" | ||
| + | # Where is the original data | ||
| + | WD_DATA_FOLDER=" | ||
| + | # Where WebDAV temporary stuff will be located | ||
| + | WD_TEMP_FOLDER=" | ||
| + | WD_ROOT_FOLDER=" | ||
| + | WD_MOUNT_FOLDER=" | ||
| + | WD_LOCKS_FOLDER=" | ||
| + | |||
| + | WD_TIMEOUT=${TIMEOUT: | ||
| + | WD_LOG_PATH="/ | ||
| + | WD_SLOT=" | ||
| + | WD_USER=${USER: | ||
| + | WD_GROUP=${GROUP: | ||
| + | |||
| + | description=${DESCRIPTION: | ||
| + | pidfile="/ | ||
| + | apache_args=( | ||
| + | -c " | ||
| + | -c " | ||
| + | -c " | ||
| + | -c " | ||
| + | -c " | ||
| + | -c "User ${WD_USER}" | ||
| + | -c "Group ${WD_GROUP}" | ||
| + | -c " | ||
| + | -c " | ||
| + | -c " | ||
| + | -c " | ||
| + | -c " | ||
| + | -c " | ||
| + | -c " | ||
| + | -c "< | ||
| + | -c " DAV On" | ||
| + | -c " AllowOverride All" | ||
| + | -c " Options -Indexes +FollowSymlinks -ExecCGI -Includes" | ||
| + | -c " Require all granted" | ||
| + | -c "</ | ||
| + | -c " | ||
| + | ) | ||
| + | |||
| + | start_pre() { | ||
| + | # script must be run with " | ||
| + | if [ " | ||
| + | then | ||
| + | ebegin " | ||
| + | eend 255 | ||
| + | return 255 | ||
| + | fi | ||
| + | # Data folder must exist: | ||
| + | if [ -z ${WD_DATA_FOLDER} -o ! -d ${WD_DATA_FOLDER} ] | ||
| + | then | ||
| + | ebegin " | ||
| + | eend 255 | ||
| + | return 255 | ||
| + | fi | ||
| + | # Create log paths | ||
| + | test -e " | ||
| + | test -e " | ||
| + | ebegin " | ||
| + | mkdir " | ||
| + | } && chown -R ${WD_USER} " | ||
| + | # Create all temporary paths: | ||
| + | for path in ${WD_TEMP_FOLDER} ${WD_ROOT_FOLDER} ${WD_MOUNT_FOLDER} ${WD_LOCKS_FOLDER} | ||
| + | do | ||
| + | test -e ${path} || { | ||
| + | ebegin " | ||
| + | mkdir -p ${path} | ||
| + | chown ${WD_USER}: | ||
| + | } | ||
| + | done | ||
| + | test -z " | ||
| + | ebegin " | ||
| + | mount -o bind ${WD_DATA_FOLDER} ${WD_MOUNT_FOLDER} | ||
| + | } | ||
| + | eend 0 | ||
| + | } | ||
| + | |||
| + | start() { | ||
| + | start-stop-daemon -w ${WD_TIMEOUT} --start --pidfile " | ||
| + | / | ||
| + | eend $? | ||
| + | } | ||
| + | |||
| + | stop_post() { | ||
| + | test -n " | ||
| + | ebegin " | ||
| + | umount ${WD_MOUNT_FOLDER} | ||
| + | } | ||
| + | eend 0 | ||
| + | } | ||
| + | </ | ||
| + | and make it executable: | ||
| <code bash> | <code bash> | ||
| - | > su - filebrowser | + | chmod +x /etc/init.d/webdav |
| - | > mkdir /data/daemons/filebrowser/ | + | |
| - | > mkdir / | + | |
| </ | </ | ||
| - | Then create the Apache | + | |
| - | < | + | === Create apache configuration files for each share === |
| - | ServerRoot | + | |
| - | LoadModule actions_module modules/ | + | By using the above init script, defining a new share means to create |
| - | LoadModule alias_module modules/mod_alias.so | + | |
| - | LoadModule auth_basic_module modules/mod_auth_basic.so | + | For our __common__ example |
| - | LoadModule authn_anon_module modules/mod_authn_anon.so | + | < |
| - | LoadModule authn_core_module modules/mod_authn_core.so | + | DESCRIPTION="Common WebDAV archive" |
| - | LoadModule authn_dbm_module modules/ | + | # this must point to where your data to be shared is located |
| - | LoadModule authn_file_module modules/ | + | DATA_FOLDER=" |
| - | LoadModule authz_core_module modules/ | + | # this will contain temporary webdav stuff, will be created if missing |
| - | LoadModule authz_dbm_module modules/ | + | TEMP_FOLDER=" |
| - | LoadModule authz_groupfile_module modules/ | + | # this refers to the URL " |
| - | LoadModule authz_host_module modules/ | + | SHARE_NAME=" |
| - | LoadModule authz_owner_module modules/ | + | GROUP=" |
| - | LoadModule authz_user_module modules/ | + | USER=" |
| - | LoadModule autoindex_module modules/ | + | PORT=10001 |
| - | < | + | |
| - | LoadModule cache_module modules/ | + | |
| - | </ | + | |
| - | LoadModule dav_module modules/ | + | |
| - | LoadModule dav_fs_module modules/ | + | |
| - | LoadModule dav_lock_module modules/ | + | |
| - | LoadModule deflate_module modules/ | + | |
| - | LoadModule dir_module modules/ | + | |
| - | LoadModule env_module modules/ | + | |
| - | LoadModule expires_module modules/ | + | |
| - | LoadModule ext_filter_module modules/ | + | |
| - | < | + | |
| - | LoadModule file_cache_module modules/ | + | |
| - | </ | + | |
| - | LoadModule filter_module modules/ | + | |
| - | LoadModule headers_module modules/ | + | |
| - | < | + | |
| - | LoadModule http2_module modules/ | + | |
| - | </ | + | |
| - | LoadModule include_module modules/ | + | |
| - | < | + | |
| - | LoadModule info_module modules/ | + | |
| - | </ | + | |
| - | LoadModule log_config_module modules/ | + | |
| - | TransferLog | + | |
| - | LoadModule logio_module modules/mod_logio.so | + | |
| - | LoadModule mime_module modules/mod_mime.so | + | |
| - | LoadModule mime_magic_module modules/ | + | |
| - | LoadModule negotiation_module modules/mod_negotiation.so | + | |
| - | LoadModule rewrite_module modules/mod_rewrite.so | + | |
| - | LoadModule setenvif_module modules/ | + | |
| - | <IfDefine STATUS> | + | |
| - | LoadModule status_module modules/ | + | |
| - | </ | + | |
| - | LoadModule unique_id_module modules/ | + | |
| - | LoadModule unixd_module modules/ | + | |
| - | < | + | |
| - | LoadModule userdir_module modules/ | + | |
| - | </ | + | |
| - | LoadModule usertrack_module modules/ | + | |
| - | LoadModule vhost_alias_module modules/ | + | |
| - | Include / | + | |
| </ | </ | ||
| + | Note the port, it needs to be unique and available. | ||
| - | Then you can create one config file for each share. This is the file for the common share **/ | + | Create |
| - | <file txt common.conf> | + | <code bash> |
| - | Include | + | ln -s /etc/init.d/webdav |
| + | </ | ||
| - | User filebrowser | ||
| - | Group users | ||
| - | DavLockDB "/ | + | === Prepare apache folders for each share === |
| - | PidFile / | + | |
| - | ErrorLog / | + | |
| - | TransferLog / | + | |
| - | CustomLog / | + | |
| - | DocumentRoot | + | The above mentioned init script will create all the needed sub-folders for you, but here is a recap: |
| + | * /share/common/webdav/root: which will map as WebDAV root (you will see why) | ||
| + | * /share/ | ||
| - | ServerName 127.0.0.1 | + | Those wll be created by the init script above if missing. They will not be deleted in any case, if existing. |
| - | Listen 127.0.0.1: | + | |
| - | < | ||
| - | DAV On | ||
| - | AllowOverride All | ||
| - | Options -Indexes +FollowSymlinks -ExecCGI -Includes | ||
| - | Require all granted | ||
| - | </ | ||
| - | SetEnv redirect-carefully | + | === Messing with the WebDAV root folder === |
| - | # vim: ts=4 filetype=apache | + | Now, the fun part is that you want to protect this behind the NGINX reverse proxy (for HTTPS and authorization reasons) and it seems that WebDAV does **not** play well with URL redirection and similar funny things. In other words, the base url you will be using on the reverse proxy **must match** the url in the Apache. You **cannot use** rewrite directives or Alias stuff. |
| - | </ | + | |
| - | Please note the Listen directive: | + | Since you will be exposing the browser-based access as **https:// |
| - | Now, the fun part is that you want to protect this behind the NGINX reverse proxy and it seems that WebDAV does **not** play well with URL redirection and similar funny things. In other words, the base url you will be using on the reverse proxy must match the url in the Apache. You cannot use rewrite directives or Alias stuff. | + | Since symbolic links cannot be used by WebDAV (could it be //that// simple?), the only viable option |
| - | Since you will be exposing the browser-based access as **https:// | ||
| - | so, create | + | === Startup Apache for the share (and autostart) === |
| + | |||
| + | Since you have already created the share specific startup script symlink and the associated config file, all you need to do is add it to the default runlevel and start it: | ||
| <code bash> | <code bash> | ||
| - | > su - filebrowser | + | rc-update add webdav.common default |
| - | > cd data/root | + | /etc/init.d/webdav.common |
| - | > mkdir webdav | + | |
| - | > cd webdav | + | |
| - | > mkdir common | + | |
| </ | </ | ||
| - | the startup script below will take take of doing the **mount | + | ===== Reverse Proxy and wrap-up ===== |
| + | Everything is protected behind the [[selfhost: | ||
| + | <file - drive.conf> | ||
| + | server { | ||
| + | server_name drive.mydomain.com; | ||
| + | listen 443 ssl; | ||
| + | listen 8443 ssl; | ||
| + | http2 on; | ||
| - | ===== Reverse Proxy ===== | + | access_log / |
| + | error_log / | ||
| - | You want to integrate all this into the SSL enabled reverse proxy, which is also using PAM authentication. | + | # WebDAV requires basic auth, while normal auth can be used with FileBrowser |
| + | include "com.mydomain/ | ||
| + | include " | ||
| - | Now, reverse proxy is simple, but this into **/etc/nginx/folders/ | + | location |
| - | <file txt filebrowser.conf> | + | include " |
| - | # Browser based access here | + | |
| - | location | + | |
| - | | + | |
| - | | + | |
| - | | + | |
| + | | ||
| - | proxy_set_header Connection $http_connection; | + | |
| - | proxy_set_header Connection ' | + | include " |
| - | proxy_cache_bypass $http_upgrade; | + | include " |
| - | | + | client_max_body_size 512M; |
| - | proxy_set_header X-Real-IP $remote_addr; | + | proxy_pass http:// |
| - | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | + | |
| - | proxy_set_header X-Forwarded-Proto $scheme; | + | proxy_set_header Connection ' |
| - | | + | proxy_cache_bypass $http_upgrade; |
| - | # WebDAV access | + | |
| - | location / | + | |
| - | # https:// | + | |
| - | # https:// | + | |
| - | set $dest $http_destination; | + | include " |
| - | if ($http_destination ~ " | + | |
| - | set $dest http:// | + | |
| + | # https:// | ||
| + | set $dest $http_destination; | ||
| + | if ($http_destination ~ " | ||
| + | set $dest http:// | ||
| + | } | ||
| + | |||
| + | # Warning: adding / at the end of the proxy_pass will break WebDAV! | ||
| + | proxy_pass http:// | ||
| + | proxy_buffering off; | ||
| + | gzip off; | ||
| + | proxy_pass_request_headers on; | ||
| + | | ||
| } | } | ||
| + | client_max_body_size 100M; | ||
| + | } | ||
| + | </ | ||
| - | proxy_pass http://127.0.0.1:10001; | + | The reverse proxy configuration doesn' |
| - | | + | |
| - | | + | This example also shows how i have integrated [[selfhost: |
| - | | + | |
| - | | + | Refer to the [[selfhost: |
| - | | + | |
| - | | + | ==== Main Directory Page ==== |
| - | | + | |
| - | | + | As you can spot from the above NGINX configuration, |
| + | |||
| + | For this i am using my [[services: | ||
| + | <file - site.json> | ||
| + | { | ||
| + | " | ||
| + | " | ||
| + | | ||
| + | | ||
| + | | ||
| + | " | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | [ { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | } ] | ||
| + | } | ||
| + | ], | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | } | ||
| } | } | ||
| </ | </ | ||
| - | and put this file include inside the usual **/ | ||
| - | ===== Wrapup and Autostart ===== | ||
| - | Create the **/ | ||
| - | <file bash fileserver.sh> | ||
| - | source / | ||
| - | BASE_PATH=/ | + | ===== Experimental stuff ===== |
| - | for i in $SHARES | + | Just some additional experiments i did, for future references. |
| - | do | + | |
| - | SHARE=$(echo $i | cut -d: -f1) | + | |
| - | PORT=$(echo $i | cut -d: -f2) | + | |
| - | OWNER=filebrowser | + | |
| - | echo Starting FileBrowser for $OWNER on share $SHARE | + | === Nephele-Serve === |
| - | su - $OWNER -c "/ | + | Replacing WebDAV with Nephele-Serve (which will support also CardDAV/CalDAV in the future) |
| - | | + | |
| - | if [ " | + | https://www.npmjs.com/package/nephele-serve |
| - | then | + | https://github.com/sciactive/nephele |
| - | echo Mounting WebDAV entry points for $SHARE | + | |
| - | mount -o bind / | + | |
| - | | + | |
| - | echo WebDAV entry point already mounted | + | |
| - | fi | + | |
| - | echo Starting WebDAV backend for $OWNER on share $SHARE | + | |
| - | su - $OWNER -c " | + | |
| - | done | + | |
| - | </file> | + | |
| - | And the usual autostart stuff: | + | NPM needs to be enabled for the fileserver user: |
| - | <file bash 40-filebrowser.start> | + | <code> |
| - | #!/bin/bash | + | NPM_PACKAGES=" |
| - | /data/ | + | mkdir -p " |
| - | </file> | + | echo " |
| + | </code> | ||
| - | Make both files executable. | + | And in **~/.bashrc**: |
| - | To access via browser: open https://<your server>/archive/common | + | < |
| + | # NPM packages in homedir | ||
| + | NPM_PACKAGES=" | ||
| + | # Tell our environment about user-installed node tools | ||
| + | PATH=" | ||
| + | # Unset manpath so we can inherit from /etc/manpath via the `manpath` command | ||
| + | unset MANPATH # delete if you already modified MANPATH elsewhere in your configuration | ||
| + | MANPATH=" | ||
| + | # Tell Node about these packages | ||
| + | NODE_PATH=" | ||
| + | </ | ||
| - | to access via WebDAV | + | Install: |
| + | <code bash> | ||
| + | source ~/ | ||
| + | npm install -g nephele-serve | ||
| + | </ | ||
| + | |||
| + | Advantages: it's a simple server that supports pam_auth. In the future, it might **also** replace [[services: | ||
| + | |||
| + | Disadvantages: | ||
| + | |||
| + | === sFtpGO | ||
| + | |||
| + | Interesting [[https://github.com/ | ||
| + | |||
| + | You need to start it once then edit **sftpgo.json**: | ||
| + | <code> | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | } | ||
| + | ], | ||
| + | </ | ||
| + | Advnatages: easier than Apache to setup, support base_url | ||
| - | Please note that using HTTP here might cause a 301 redirect to HTTPS, | + | Disadvantages: |
| + | === KaraDAV / PicoDAV === | ||
| + | [[https:// | ||
| + | [[https:// | ||
| + | Unfortunately, | ||