Unbound is a modern DNS server which is capable of resolving and forwarding your requests using DoT and DoH. I will show you how to use Unbound for your home network using DoT for upstream and DoH for downstream. Given that DoH is not common on PCs, the classic good old DNS on port 53 (both UDP and TCP) will also be available.

It's very simple to setup on Gentoo (see here) and it also support DNSSEC (which, at this time, i have not configured yet).

So, first of all enable DNSCrypt for Unbound by creating the file /etc/package.use/unbound:

unbound

net-dns/unbound dnscrypt

then emerge Unbound:

emerge unbound

I am using the following /etc/unbound/unbound.conf:

unbound.conf

server:
        verbosity: 1
        num-threads: 2
        interface: 10.0.0.1@53     # Listen to home interface
        interface: 127.0.0.1@53   #  and listen to localhost as well
        interface: 10.0.0.1@853     # Listen to home interface for DoT
        interface: 127.0.0.1@853   #  and listen to localhost as well for DoT
        interface: 127.0.0.1@4443 # listen for DoH on local only port 
        port: 53           
        https-port: 4443
        http-notls-downstream: yes 
        so-reuseport: yes
        cache-min-ttl: 300
        cache-max-ttl: 86400
        do-ip4: yes
        do-ip6: yes
        do-udp: yes
        do-tcp: yes
        use-systemd: no
        do-daemonize: yes
       # For security reasons, only clients on the home interface can use the DNS service        
       access-control: 10.0.0.0/24 allow   
        access-control: 127.0.0.1/8 allow     # and, of course, localhost as well
        use-syslog: yes
        hide-identity: yes
        hide-version: yes
        harden-short-bufsize: yes
        harden-large-queries: yes
        harden-glue: yes
        harden-dnssec-stripped: yes
        harden-below-nxdomain: yes
        harden-referral-path: yes
        harden-algo-downgrade: yes
        qname-minimisation: yes
        qname-minimisation-strict: no
        aggressive-nsec: yes
        use-caps-for-id: yes
        prefetch: yes
        rrset-roundrobin: yes
        minimal-responses: yes
        # This will enable DoT (upstream)
        tls-cert-bundle: "/etc/ssl/certs/ca-certificates.crt"
        
        # This will add Ad blocking
        include: /etc/unbound/adservers.conf
        include: /etc/unbound/local.conf
        
remote-control:
        control-enable: yes

forward-zone:
        name: "."
        # Use Google DNS as upstream DNS (put here your preferred ones if not Google)
        forward-tls-upstream: yes
        forward-addr: 8.8.8.8@853
        forward-addr: 8.8.4.4@853

To configure specific _internal_ hosts, you need to define a local-zone and a matching local-data rows as defined above. You might want to move these information to specific files to include (like the adservers.conf) for easier maintenance if you have lots of internal names.

At this point, read the Ads Blocking page to create the /etc/unbound/adservers.conf file before continuing.

You need to populate your /etc/resolv.conf with Unbound as the default nameserver:

resolv.conf

# My own local DNS resolver (Unbound)
nameserver 127.0.0.1

Don't forget to autostart Unbound service:

rc-update add unbound default
/etc/init.d/unbound start

NOTE: as far as i managed to understand, DoT is always enabled on the upstream connection only. You do not need, not want, DoT within your home network.

The local.conf will contain your own home local addresses:

local.conf

server:
       # Add a local resolve for the home server
        local-zone: "home.mydomain.com." redirect
        local-data: "home.mydomain.com. A 10.0.0.1"

The above configuration already enable DoH (DNS over HTTPS) in Unbound. I set it up on port 4443 because port 443 is already taken by the NGINX reverse proxy, of course, so you will need to create a reverse proxy from NGINX to Unbound covering the endpoint /dns-query.

Drop the following configuration file into your NGINX home server configuration (see The Reverse Proxy concept for more details):

dns.conf

location /dns-query {
        if ( $request_method !~ ^(GET|POST|HEAD)$ ) {
                return 501;
        }
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect off;
        proxy_buffering off;
        grpc_pass grpc://127.0.0.1:4443;
}

Note that the classic proxy_pass cannot be used because it does not support HTTP2 upstream, which is mandatory for Unbound, so you need to use grpc_pass and disble HTTPS in Unbound, as it's been done in the above configuration files. More details here.

Restart your NGINX, and your DoH should be operative.

Please keep in mind that DoH only works with domain names due to how certificates work, IP addresses won't work.

Just to be clear: this DNS over HTTPS is a huge mess and a great headache. In fact, it's more a shitty way to prevent people to run DNS-based ad-blockers than anything. DoT already provided everything needed, but DoH is actually in the hands of the “big players” that can bypass any home-network security in this way, because HTTPS traffic cannot be filtered.