File Server

In order to access your files from everywhere you need the following access vectors:

• From a web browser (to access anywhere)
• Via WebDAV (to access from apps and phone)
• Using NFS (to access from Linux)
• Using SMB (to access from Windows)

It is not possible to achieve all this using one single tool, so you will need to leverage different pieces together, and i will show yo how.

The idea is to create one share area where your users will be able to store files. It is possible to extend this idea also to user-specific areas where each user can put private stuff not visible by other users, but this require a little bit extra complexity and might be addressed in the future.

You will be using your home server authentication, there will be no need to create new users anywhere, and it will of course be protected by the Reverse Proxy for external access.

I will show you some DIY glue to manage everything together.

Let's assume you will need one common share, called with lots of imagination common, and the files will be under /home/common (for example).

Create a text file under /etc/conf.d/shares to manage all your shares:

shares

SHARES="common:3002 other:3003"

(as an example, i defined also a second share called other)

where “common” and “other” is the name of the folder under /home and 3002/3003 are the ports number (which will be needed for NGINX reverse proxy access via browser) for each share.

All users which need to access the shares must be in the users group: the common share will be accessible by any user in the users group.

You will also need to add a specific fileserver user to run the associated services, then go ahead and create the /home/common folder. You need to assign that folder to the users group and the fileserver user:

useradd -d /data/services/fileserver -m fileserver -g users
mkdir /home/common
chown fileserver:users  /home/common

I strongly recomend to use FileBrowser because it's lightweight, don't get in the way, is flexible and simple to use. Check the linked page for the generic installation instructions, and here are the specific details for this case.

FileBrowser will run as the fileserver user that you created above. You will need to create the following folders architecture in your *fileserver* home folder: - bin: where the FileBrowser binary will be located - data/db: where the FileBrowser databases files will be stored - data/logs: where the various log files will be created

You need to set the umask for the user to 0002 so that any new files created by it will be writable by the users.

I assume you have installed FileBrowser as indicated above (you should skip the Reverse Proxy instructions that will be specified later on here). Then:

su - fileserver
echo "umask 0002" >> ~/.bashrc
source ~/.bashrc
mkdir data data/logs data/db

Now, you will need to start a copy of FileBrowser for each share you want to have, and it must be owned by the user that want file permissions on that share. To achieve this, you will be using a special script called fileserver.sh which i will show you at the end, because it will contain also the WebDAV start stuff in it.

While there are a few WebDAV servers like Dave, they seems to be either unmaintained or overly complicated.

Also NGINX can be a WebDAV server, but it seems to be buggy and not supporting LOCK stuff, so i decided to go with Apache web server, which also has a long standing WebDAV implementation.

The idea here is to run a dedicated copy of Apache as user fileserver and group users so that it can access and manage the shared files. So first you need to emerge apache:

emerge apache`

WebDAV is enabled by default in Gentoo Apache ebuild, so need to fix USE flags.

Running apache manually, and not as a system service, requires some effort, so, buckle up!

First of all, Apache needs some folders to operate, so you need to create:

• /data/daemons/fileserver/data/conf: to store the apache config file
• /data/daemons/fileserver/data/roots: which will map as WebDAV root (you will see why)
• /data/daemons/fileserver/data/locks: which will be used for WebDAV lock databases
• /data/daemons/fileserver/data/pids: which will be used to store apache PID files

su - fileserver
mkdir /data/daemons/fileserver/data/conf
mkdir /data/daemons/fileserver/data/root
mkdir /data/daemons/fileserver/data/locks
mkdir /data/daemons/fileserver/data/pids

Then create the global Apache config file for all the shares. You should create this config that will be used by each share /data/daemons/fileserver/data/conf/apache_global.conf:

apache_global.conf

ServerRoot "/usr/lib64/apache2"
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
<IfDefine CACHE>
LoadModule cache_module modules/mod_cache.so
</IfDefine>
LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule dav_lock_module modules/mod_dav_lock.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule ext_filter_module modules/mod_ext_filter.so
<IfDefine CACHE>
LoadModule file_cache_module modules/mod_file_cache.so
</IfDefine>
LoadModule filter_module modules/mod_filter.so
LoadModule headers_module modules/mod_headers.so
<IfDefine HTTP2>
LoadModule http2_module modules/mod_http2.so
</IfDefine>
LoadModule include_module modules/mod_include.so
<IfDefine INFO>
LoadModule info_module modules/mod_info.so
</IfDefine>
LoadModule log_config_module modules/mod_log_config.so

# This is needed to avoid error on load due to default path being not accessible
TransferLog /data/daemons/fileserver/data/logs/common_transfer_log

LoadModule logio_module modules/mod_logio.so
LoadModule mime_module modules/mod_mime.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
<IfDefine STATUS>
LoadModule status_module modules/mod_status.so
</IfDefine>
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule unixd_module modules/mod_unixd.so
<IfDefine USERDIR>
LoadModule userdir_module modules/mod_userdir.so
</IfDefine>
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
Include /etc/apache2/modules.d/*.conf

Then you can create one config file for each share. This is the file for the common share /data/daemons/fileserver/data/conf/common.conf:

common.conf

Include /data/daemons/fileserver/data/conf/apache_global.conf

User fileserver
Group users

DavLockDB "/data/daemons/fileserver/data/locks/common"
PidFile /data/daemons/fileserver/data/pids/common.pid
ErrorLog /data/daemons/fileserver/data/logs/common_error_log
TransferLog /data/daemons/fileserver/data/logs/common_transfer_log
CustomLog /data/daemons/fileserver/data/logs/common_access_log common

DocumentRoot /data/daemons/fileserver/data/roots

ServerName 127.0.0.1
Listen 127.0.0.1:10001

<Directory /data/daemons/fileserver/data/roots>
    DAV On
    AllowOverride All
    Options -Indexes +FollowSymlinks -ExecCGI -Includes
    Require all granted
</Directory>

SetEnv redirect-carefully

# vim: ts=4 filetype=apache

Please note the Listen directive: you want apache to be bound to 127.0.0.1 only and note the port too, this port will be needed for the reverse proxy. Each share will need it's own port.

Now, the fun part is that you want to protect this behind the NGINX reverse proxy (for HTTPS and authorization reasons) and it seems that WebDAV does not play well with URL redirection and similar funny things. In other words, the base url you will be using on the reverse proxy must match the url in the Apache. You cannot use rewrite directives or Alias stuff.

Since you will be exposing the browser-based access as https://your_server/archive/common and the WebDAV access as https://your_server/webdav/common it means that we need to connect your /home/common folder to /data/daemons/fileserver/data/roots/webdav/common for it to work. Since symbolic links cannot be used by WebDAV (could it be that simple?), the only viable option is mount -o bind which needs to be done by root.

so, create the paths first:

 su - fileserver
cd /data/daemons/fileserver/data/root
mkdir webdav
cd webdav
mkdir common

the startup script below will take take of doing the mount -o bind which is mandatory for WebDAV to work.

You want to integrate all this into the SSL enabled reverse proxy, which is also using PAM authentication.

Now, reverse proxy is simple, but this into /etc/nginx/folders/fileserver.conf:

fileserver.conf

  # Browser based access here
  location /archive/common/ {
        client_max_body_size 512M;

        proxy_pass http://127.0.0.1:3002; # Note this port! Match the port used by Apache here!
        proxy_http_version 1.1;

        proxy_set_header Connection $http_connection;
        proxy_set_header Connection 'upgrade';
        proxy_cache_bypass $http_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    # WebDAV access
    location /webdav/common {
        # https://mailman.nginx.org/pipermail/nginx/2007-January/000504.html - fix Destination: header
        # https://trac.nginx.org/nginx/ticket/348 - bug, workaround with named capture
        set $dest $http_destination;
        if ($http_destination ~ "^https://(?<myvar>(.+))") {
                set $dest http://$myvar;
        }

        proxy_pass http://127.0.0.1:10001;
        proxy_redirect off;
        proxy_buffering off;
        gzip off;
        proxy_pass_request_headers on;
        proxy_set_header Destination       $dest;
        proxy_set_header      Host $host;
        proxy_set_header      X-Real-IP $remote_addr;
        proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
}

refer to the The Reverse Proxy concept page to activate this specific NGIX configuration.

This is the easiest part, you will simply use Gentoo's default NFS and Samba packages with minimal configuration.

So, emerge them and add the main services at boot:

emerge -v  net-fs/samba net-fs/nfs-utils
rc-update add default nfs
rc-update add default samba

Then setup the NFS share editing /etc/exports:

exports

/home/common 10.0.0.0/24(rw,async,no_subtree_check)

And start the NFS service:

/etc/init.d/nfs start

NFS export will probably mess a bit up on files ownership due to how Linux matches locla users with remote users, YMMV.

Edit the samba config file under /etc/samba/smb.conf to your liking, pay attention to the following lines:

 ... omissis ...
[global]
   workgroup = HOME
   server string = Samba Server
   security = user
   hosts allow = 10.0.0.
 ... omissis ...
[common]
   comment = Common Area
   path = /home/common
   valid users = <users list>
   public = yes
   guest ok = yes
   writable = yes
   printable = no

Samba requires to specify a list of allowed users.

Now it's time to ensure everything starts properly at boot, so create the /data/daemons/filserver/fileserver.sh script and make it executable:

fileserver.sh

source /etc/conf.d/shares

BASE_PATH=/data/daemons/filebrowser/data

for i in $SHARES
do
        SHARE=$(echo $i | cut -d: -f1)
        PORT=$(echo $i | cut -d: -f2)
        OWNER=filebrowser

        echo Starting FileBrowser for $OWNER on share $SHARE
        su - $OWNER -c "/data/daemons/filebrowser/bin/filebrowser config set --auth.method=noauth -d $BASE_PATH/db/filebrowser_$SHARE.db >/dev/null"
        su - $OWNER -c "/data/daemons/filebrowser/bin/filebrowser -r /deposito/$SHARE -p $PORT -b /archive/$SHARE -d $BASE_PATH/db/filebrowser_$SHARE.db -l $BASE_PATH/logs/filebrowser_$SHARE.log 2> $BASE_PATH/logs/filebrowser_${SHARE}_run.log"&

        if [ "$(mount | grep /deposito/daemons/filebrowser/data/roots/webdav/$SHARE)" = "" ]
        then
            echo Mounting WebDAV entry points for $SHARE
            mount -o bind /home/$SHARE /data/daemons/filebrowser/data/roots/webdav/$SHARE
        else
            echo WebDAV entry point already mounted
        fi
        echo Starting WebDAV backend for $OWNER on share $SHARE
        su - $OWNER -c "apache2 -f /data/daemons/filebrowser/data/conf/$SHARE.conf"
done

And the usual autostart stuff /etc/loca.d/40-fileserver.start, and make also this one executable:

40-filserver.start

#!/bin/bash
/data/daemons/filebrowser/fileserver.sh

To access via browser: https://10.0.0.1/archive/common

to access via WebDAV clients: https://10.0.0.1/webdav/common

Please note that using HTTP here might cause a 301 redirect to HTTPS, and WebDAV clients will fail. So use HTTPS URL in webdav clients.

Just some additional experiments i did, for future references.

Replacing WebDAV with Nephele-Serve (which will support also CardDAV/CalDAV in the future)

https://www.npmjs.com/package/nephele-serve https://github.com/sciactive/nephele

NPM needs to be enabled for the fileserver user:

NPM_PACKAGES="$HOME/.npm-packages" 
mkdir -p "$NPM_PACKAGES"  
echo "prefix = $NPM_PACKAGES" >> ~/.npmrc

And in ~/.bashrc:

# NPM packages in homedir
NPM_PACKAGES="$HOME/.npm-packages"
# Tell our environment about user-installed node tools
PATH="$NPM_PACKAGES/bin:$PATH"
# Unset manpath so we can inherit from /etc/manpath via the `manpath` command
unset MANPATH # delete if you already modified MANPATH elsewhere in your configuration  
MANPATH="$NPM_PACKAGES/share/man:$(manpath)"
# Tell Node about these packages
NODE_PATH="$NPM_PACKAGES/lib/node_modules:$NODE_PATH"

Install:

source ~/.bashrc  
npm install -g nephele-serve

Advantages: it's a simple server that supports pam_auth. In the future, it might also replace Radicale with a single service.

Disadvantages: do not support base_url, so unable to host under /webdav even with reverse proxy.

Interesting sFtpGo support both web-browser access and WebDAV from a single tool.

You need to start it once then edit sftpgo.json:

"external_auth_hook": "/data/daemons/fileserver/login.sh",
"webdavd": {
    "bindings": [
      {
        "port": 10001, 
        "address": "127.0.0.1",
        "enable_https": false,
        "certificate_file": "",
        "certificate_key_file": "",
        "min_tls_version": 12,
        "client_auth_type": 0,
        "tls_cipher_suites": [],
        "prefix": "/webdav/common",
        "proxy_allowed": [],
        "client_ip_proxy_header": "",
        "client_ip_header_depth": 0,
        "disable_www_auth_header": false
      }
    ],

Advnatages: easier than Apache to setup, support base_url

Disadvantages: cannot use pam_auth and cannot disable authentication. So double-auth over reverse proxy that might be annoying.