The Reverse Proxy concept

The use of a reverse proxy is the key at the foundation of ensuring security, isolation and flexibility in accessing your self-hosted services.

A reverse-proxy is a web server that sits in the middle and handles all requests toward your services adding, on top, layers of encryption (HTTPS/SSL), authentication, load-balancing and security. If your services are properly written (not too many, but the best ones are) they will accept your SSO authentication directly without even the need to create users for each service, in this case your reverse-proxy will also cater for your SSO (Single Sign On) solution . More on this on the dedicated page Single Sign On, but keep reading this page first.

The reverse-proxy will take care of handling HTTPS/SSL certificates in one centralized place making it much easier to configure all your services without HTTPS then converting seamlessly all the HTTP traffic to HTTPS. It's much easier to manage all the certificates in one place rather than depending on each service capability to handle HTTPS independently.

Also, using a well known, solid and proven web server will alleviate the risk that each service might expose a poorly written, non-scalable or worse, internal web server to end users.

And as a final note, using a reverse-proxy you can organize easily all your services either under one single domain or with sub-domains, according to your specific needs.

My choice for a web server in this case is NGINX between the many available as Open Source because:

• It's much easier than Apache to setup as a reverse-proxy, also less resource hungry, and works with more SSOs than Apache.
• It has more features than Caddy.
• It is fully integrated in Let's Encrypt SSL infrastructure / CertBot script

In general NGINX is fully featured but still very lightweight and secure HTTP server that shines as reverse-proxy. If you need to add more features, like PHP support or FastCGI, NGINX will support you without the need for an additional service on your server.

There are two different philosophies on how to host services: serve as a sub-path of a domain, or use sub-domains. I used to like best the sub-path approach, but indeed a good mix of the two ways is preferable.

Let's assume you have your own domain mydomain.com and you want to expose a service called jellyfin (a well known media-server). You can expose it:

• As a sub-path: https://mydomain.com/jellyfin
• As a sub-domain: https://jellyfin.mydomain.com

Here are the main points and drawbacks of each solution.

As a sub-path:

• Pros: only one domain needed, no need to create sub-domains
• Pros: the service existence is unknown to anybody not authorized
• Cons: each service must support Base URL setting (well, not all do!)
• Cons: SSO support must be somehow consistent to avoid headaches (well, SSO support is still spotty today!)
• Cons: security wise, cookies and CORS can bring unintended vulnerabilities between services, because they all share the same subdomain.
• Cons: all services share the same HTTPS certificate.

As a sub-domain:

• Pros: any service will work, no need to support Base URL
• Pros: each service can have it's own HTTPS certificate
• Pros: each service is neatly organized in it's own subdomain
• Pros: cookies are not shared between services, and CORS protection works
• Cons: exposed to public knowledge (DNS records are public) that the service exist
• Cons: also public knowledge because there are services indexing all existing certificates.

Note: you can create wildcard certificates that will match any subdomain, but there are drawbacks to this and it's not a good idea, security wise. You can still mitigate the one certificate per subdomain by adding each subdomain to the same certificate of course, but you will still need to extend your certificate each time you add a subdomain: this is my approach.

To make a story short, i go with subdomains for well separated services, while going with sub-paths when sharing stuff that kind belongs together. Also, a deciding factor is whether the selected services do support SSO properly or not.

The reverse proxy is installed on the local server, you should have already guessed that remote access is performed using the SSH tunneling described in the specific page. The underlying idea is that you will have your reverse proxy listening to different ports, and these ports will be forwarded to your external server using the SSH tunnels. Differentiating the ports is required to be able to apply SSO authentication depending on where your user connects from.

The setup i am describing uses three different ports:

• Port 80: both to local and remote, will just be a redirect to HTTPS
• Port 443: standard HTTPS for internal access, mostly to bypass SSO authentication
• Port 8443: HTTPS with SSO authentication for external access

Note: for Let's Encrypt CertBot to work properly you need to redirect both port 80 and 443 from your external server to your internal server. CertBot will shutdown your NGINX and spin a custom NGINX server that you cannot tweak so it's critical that your SSH tunnels are properly forwarding ports 80 and 443 from the external server to the internal one, or it will not work.

NGINX installation on the home server is pretty straightforward, but you need to enable some specific modules:

• auth_request: needed for SSO like authelia
• auth_pam: needed for PAM SSO
• sub is used to allow substitutions inside the pages proxied, to fix web applications that don't play well with reverse-proxies
• gunzip is used to unzip the requests and let the sub module works also on compressed requests
• realip is needed by SSO like authelia

While NGINX support WebDAV, i strongly suggest you dont enable it as you will not be using it. Apache WebDAV support is much better.

So create the file /etc/portage/package.use/nginx with the following lines:

nginx

app-misc/mime-types nginx
www-servers/nginx NGINX_MODULES_HTTP: auth_request auth_pam dav dav_ext gunzip sub realip xslt 

Note: you might want to tweak the second line to your needs, see the flags for nginx and adapt.

Now install nginx:

emerge -v nginx

You can start it after you have configured it.

There are many ways to write nice NGINX config files, i will show you mine which i find quite effective, organized and simple. It make use of the import directive and splits the configuration to at least one file per service and one file per sub-domain.

Assumptions:

• Your domain is mydomain.com, and it has a static landing page under /var/www/html/index.html
• Your service X is reachable under https://mydomain.com/serviceX (subpath)
• Your service Y is reachable under https://y.mydomain.com (subdomain)
• All HTTP traffic is redirected to HTTPS

The top-level mydomain.com will have it's own folder, then you will create a set of sub-folders stemming from the main domain, one folder for each sub-domains, and inside each folder one configuration file for each sub-path served on that sub-domain.

So you will need the following files:

• /etc/nginx/nginx.conf: main config file, entry point.
• /etc/nginx/com.mydomain/certbot.conf: SSL certificates configuration for mydomain.com
• /etc/nginx/com.mydomain/mydomain.conf: global config for mydomain.com
• /etc/nginx/com.mydomain/serviceX.conf: config for serviceX on mydomain.com
• /etc/nginx/com.mydomain/y/y.conf: config for serviceY on y.mydomain.com
• plus any other SSO specific config files.

The certbot.conf file will be created later on, the specific SSO config files are described in the Authentication page.

So, here is the content for the main /etc/nginx/nginx.conf:

nginx.conf

user nginx nginx;

error_log /var/log/nginx/error_log info;

events {
        worker_connections 1024;
        use epoll;
}

http {
        include /etc/nginx/mime.types;
        # Unknown stuff is considered to be binaries
        default_type application/octet-stream;
        # Set a reasonably informing log format
        log_format main
                '$remote_addr - $remote_user [$time_local] '
                '"$request" $status $bytes_sent '
                '"$http_referer" "$http_user_agent" '
                '"$gzip_ratio"';
        # Improve file upload to client by avoiding userspace copying
        tcp_nopush on;
        sendfile on;
        # Indexes are html by default
        index index.html;

        # General catch-all for HTTPS redirection, we don't like serving plain HTTP
        server {
                listen 80 default_server;
                return 301 https://$host$request_uri;
        }

        # Using Authelia SSO can lead to longer headers, better increase buffers
        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 128;

        # Add domains here (only the main config file for each domain!)
        include com.mydomain/mydomain.conf;
}

This will set your defaults for every service and site served by this reverse proxy, then will load the mydomain.com specific configuration file.

Now, for the specific mydomain.com, you need the following config file under /etc/nginx/com.mydomain/mydomain.conf:

mydomain.conf

access_log /var/log/nginx/mydomain.com_access_log main;
error_log /var/log/nginx/mydomain.com_error_log info;

# simple catch-all server for the domain
server {
       # respond both to local, internal, IP directly and to mydomain.com
        server_name 10.0.0.1 mydomain.com;
        # Port for users from outside
        listen 8443 ssl;
        # Port for users from inside
        listen 443 ssl;
        http2 on;

        # unauthenticated static landing page (maybe a "get off my lawn" GIF...)
        location / {
               root /var/www/html;
        }

       # include all sub-paths for mydomain.com:
       include serviceX.conf
}

# include all sub-domains entry points:
include com.mydomain/y/y.conf;

# include HTTPS certs stuff:
include com.mydomain/certbot.conf;

This will create the basic setup for your base domain name. I have assumed you want a static landing page, but you might put a redirect to service Y or service X… Or add a dashboard, of course protected by your SSO…

It should be clear now that each sub-domain will have it's own sub-folder and contain at least one (or more) configuration files inside for each sub-path, like the one for serviceY.

I will assume that serviceY perform it's own authentication and cannot use SSO:

y.conf

server {
        server_name y.mydomain.com;
        listen 8443 ssl; # external access
        listen 443 ssl; # internal access
        access_log /var/log/nginx/y.mydomain.com_access_log main;
        error_log /var/log/nginx/y.mydomain.com_error_log info;
        location / {
                #Generic proxy pass to proxied service
                proxy_pass http://127.0.0.1:8000;
        }
}

I suggest you split all sub-paths for each sub-domain in a separate config file and include them inside the server block, like i did above for mydomain.com.

In my setup i have some differences when a service is accessed from within the home network, or from outside the home network.

The key point is that external access comes trough port 8443, while internal aces comes trough port 443. This allows you to differentiate your setup with server blocks.

So, for example, a service _only_ available inside the home network will have something like:

server {
        server_name internal_only.mydomain.com;
        listen 443 ssl; # internal access
        http2 on;
        access_log /var/log/nginx/internal_only.mydomain.com_access_log main;
        error_log /var/log/nginx/inernal_only.mydomain.com_error_log info;
        location / {
                #Generic proxy pass to proxied service
                proxy_pass http://127.0.0.1:8000;
        }
}

While a service that can be accessed both from internal and external:

server {
        server_name serviceZ.mydomain.com;
        listen 8443 ssl; # external access
        listen 443 ssl; # internal access
        http2 on;
        access_log /var/log/nginx/serviceZ.mydomain.com_access_log main;
        error_log /var/log/nginx/serviceZ.mydomain.com_error_log info;
        location / {
                #Generic proxy pass to proxied service
                proxy_pass http://127.0.0.1:8000;
        }
}

A service where you want to differentiate between internal and external, for example adding SSO authentication only for external access:

server {
        server_name serviceZ.mydomain.com;
        listen 443 ssl; # internal access
        http2 on;
        access_log /var/log/nginx/serviceZ.mydomain.com_access_log main;
        error_log /var/log/nginx/serviceZ.mydomain.com_error_log info;
        location / {
                #Generic proxy pass to proxied service
                proxy_pass http://127.0.0.1:8000;
        }
}
server {
        server_name serviceZ.mydomain.com;
        listen 8443 ssl; # external access
        http2 on;
        [[[ put here your SSO lines ]]]
        access_log /var/log/nginx/serviceZ.mydomain.com_access_log main;
        error_log /var/log/nginx/serviceZ.mydomain.com_error_log info;
        location / {
                #Generic proxy pass to proxied service
                proxy_pass http://127.0.0.1:8000;
        }
}

In this case, you can even optimize more by moving the location lines, which are identical, inside another file that you include twice. Better to avoid redundancy!

Of course, refer to the SSI page for more details on SSO.

Nowadays HTTPS is a must for many reasons, including privacy and security. I assume this is a mandatory requirement. A lot of services will not even work without HTTPS.

Enabling HTTPS requires the generation of valid SSL certificates for your domain(s). You can do that with self-signed certificates but that will still flag as insecure on your browser and some client apps might even not work properly. A better solution is to use the Let's Encrypt certification authority which is an open-source, public and free Certificate Authority that let's you generate and manage your certificates.

How does it work?

first of all:

1. You ask Let's Encrypt to create a certificate for each one of your sub-domains (automated by CertBot)
2. You setup the certificate (automated by CertBot)
3. You renew periodically the certificate (automated by CertBot)

Then:

1. You connect with browser to https://mydomain.com
2. Your server provide the certificate
3. Your browser verify that the certificate is valid against the Let's Encrypt Root Certificate
4. You are good to go!

Using self-signed certificates works too, but since for the browser to validate the certificate needs to already know the associated Certificate Authority, the site will still appear as untrusted. Since Let's Encrypt is A nonprofit Certificate Authority providing TLS certificates with the mission to provide everybody with security and trust, there is no reason not to use it.

Luckly, Let's Encrypt provides a neat software called CertBot that can automate all the steps for the major web servers, including NGINX. CertBot will send requests to Let's Encrypt, spin up an NGINX server for you and store the certificate. The only thing you need to do is including the proper config file into NGINX and restart it.

Install CertBot and the NGINX plugin:

 > emerge -v certbot-nginx certbot

This will pull in all the required software to perform the exchange with Let's Encrypt infrastructure. At this point you only need to run Certbot to generate a certificate for your external domain name:

 > certbot --nginx certonly -d mydomain.com -d y.mydomain.com -d xxxx

Now, you must generate certificates that chains toghether all the subdomains you use. This means that if you add, later on, another sub-domain to host a new service you will need to re-run the above certbot command adding -d newsubdomain.mydomain.com. And do not forget all the older ones! Luckly, domain names can be chained to on single certificate, so you do not have to edit your NGINX config ever again for CertBot to work.

Put this content into your /etc/nginx/com.mydomain/certbot.conf:

certbot.conf

ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

Of course, adapt the paths for your specific case.

Let's Encrypt certificates last 90 days, then they need to be renewed. This is automated by CertBot but you need to call it periodically. You can use crontab for this. Edit root crontab:

crontab -e

and write the following lines:

47 5 * * * certbot renew  &>> /var/log/certbot.log
31 16 * * * certbot renew  &>> /var/log/certbot.log

there you go!

You can now start your nginx server:

rc-update add nginx default
/etc/init.d/nginx start

When you need to add a new subbomain to your certificate, you can copy (and adapt) the following script i use:

certbot_script.sh

#!/bin/bash

DOMAINS="mydomain.con y.mydomain.com other.mydomain.com"

domains=
for i in ${DOMAINS}
do
        domains="${domains} -d ${i}"
done

certbot certonly --expand --nginx ${domains}

So FIRST you update the script adding the new domain at the end of the DOMAINS line, then you run the script and restart your NGINX.

To be able to run system scripts and, in general, CGIs on NGINX you need to do some additional configuration. NGINX is not capable of running CGI scripts at all. It has only support for FastCGI protocol, which is quite different and not directly compatible with standard CGI.

For using CGI directly with NGINX (another option could be to run Apache or another web server in addition, but why?) you can install and setup fcgiwrap and it's companion spawn package:

emerge www-misc/fcgiwrap www-servers/spawn-fcgi

Spawn-fcgi allows you to run one instance of fcgiwrap for each service you need to run. This is an excellent approach to keep services separated and each one in it's own user.

Since you want to run fcgiwrap set up like this:

• Setup your spawn-fcgi config file in /etc/conf.d/spawn-fcgi.fcgiwrap
• Create a start script in /etc/init.d/spawn-fcgi.my-cgi.

The contents of the config file sohuld be:

spawn-fcgi.fcgiwrap

# The "-1" is added on my system, check on your YMMV!
FCGI_SOCKET=/var/run/fcgiwrap.sock-1
FCGI_PORT=
# The -f send stderr to nginx log
FCGI_PROGRAM="/usr/sbin/fcgiwrap -f"
FCGI_USER=nginx
FCGI_GROUP=nginx
FCGI_EXTRA_OPTIONS="-M 0700"
ALLOWED_ENV="PATH"

And to do all the above:

cp /etc/conf.d/spawn-fcgi /etc/conf.d/spawn-fcgi.fcgiwrap
 ln -s /etc/init.d/spawn-fcgi /etc/init.d/spawn-fcgi.fcgiwrap
rc-update add spawn-fcgi.fcgiwrap default
/etc/init.d/spawn-fcgi.fcgiwrap start

Then enable it in your NGINX config by adding the following directives

cgi.conf

       location /my_cgi {
            fastcgi_param DOCUMENT_ROOT /path/to/gci/executable/folder/;
            fastcgi_param SCRIPT_NAME   my_cgi;
            fastcgi_pass unix:/var/run/fcgiwrap.sock;
       }

Assuming you want to add a new service to your Reverse Proxy and the relative configuration has been written in service.conf file, you need to include it inside your URL's configuration file. If the service needs to be under https://mydomain.com you will need to add it like:

include "com.mydomain/service.conf";

and then restart nginx:

/etc/init.d/nginx restart