External Pi-hole with IPv6 – Setup a secured Pi-hole DNS service on Docker using Linode/AWS

Let me address the question of why I decided to put a DNS server (Pihole) exposed to the internet (not fully open but still).

I needed/wanted to set up an Umbrella/NextDNS/CF type DNS server that’s publicly accessible but secured to certain IP addresses.

Sure NextDNS is an option and its cheap with similar features, but i wanted roll my own solution so i can learn a few things along the way

I can easily set this up for my family members with minimal technical knowledge and unable to deal with another extra device (Raspberry pi) plugged into their home network.

This will also serve as a quick and dirty guide on how to use Docker compose and address some Issues with Running Pi-hole, Docker with UFW on Ubuntu 20.x

So lets get stahhhted…….

Scope

  • Setup Pi-hole as a docker container on a VM
  • Enable IPV6 support
  • Setup UFW rules to prune traffic and a cronjob to handle the rules to update with the dynamic WAN IPs
  • Deploy and test

What we need

  • Linux VM (Ubuntu, Hardened BSD, etc)
  • Docker and Docker Compose
  • Dynamic DNS service to track the changing IP (Dyndns,no-Ip, etc)

Deployment

Setup Dynamic DNS solution to track your Dynamic WAN IP

for this demo, we are going to use DynDNS since I already own a paid account and its supported on most platforms (Routers, UTMs, NAS devices, IP camera-DVRs, etc)

Use some google-fu there are multiple ways to do this without having to pay for the service, all we need is a DNS record that's up-to-date with your current Public IP address. 

For Network A and Network B, I’m going to use the routers built-in DDNS update features

Network A gateway – UDM Pro

Network B Gateway – Netgear R6230

Confirmation

Setup the VM with Docker-compose

Pick your service provider, you can and should be able to use a free tier VM for this since its just DNS

  • Linode
  • AWS lightsail
  • IBM cloud
  • Oracle cloud
  • Google Compute
  • Digital Ocean droplet

Make sure you have a dedicated (static) IPv4 and IPv6 address attached to the resource

For this deployment, I’m going to use a Linode – Nanode, due to their native IPv6 support and cause I prefer their platform for personal projects

Setup your Linode VM – Getting started Guide

SSH in to the VM or use weblish console

Update your packages and sources

sudo apt-get update 
install Docker and Docker Compose

Assuming you already have SSH access to the VM with a static IPv4 and IPv6 address

Guide to installing Docker Engine on Ubuntu

Guide to Installing Docker-Compose

Once you have this setup confirm the docker setup

docker-compose version

Setup the Pi-hole Docker Image

Lets Configure the docker networking side to fit our Needs

Create a Seperate Bridge network for the Pi-hole container

I guess you could use the default bridge network, but I like to create one to keep things organized and this way this service can be isolated from the other containers I have

docker network create --ipv6 --driver bridge --subnet "fd01::/64" Piholev6

verification

We will use this network later in docker compose

With the new ubuntu version 20.x, Systemd will start a local DNS stub client that runs on 127.0.0.53:53

which will prevent the container from starting. because Pi-hole binds to the same port UDP 53

we could disable the service but that breaks DNS resolution on the VM causing more headaches and pain for automation and updates

After some google fu and trickering around this this is the workaround i found.

  • Disable the stub-listener
  • Change the symlink to the /etc/resolved.conf to /run/systemd/resolve/resolv.conf
  • push the external name servers so the VM won’t look at loopback to resolve DNS
  • Restart systemd-resolved
Resolving Conflicts with the systemd-resolved stub listener

We need to disable the stub listener thats bound to port 53, as i mentioned before this breaks the local dns resolution we will fix it in a bit.

sudo nano /etc/systemd/resolved.conf

Find and uncomment the line “DNSStubListener=yes” and change it to “no”

After this we need to push the external DNS servers to the box, this setting is stored on the following file

/etc/resolv.conf
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53

But we cant manually update this file with out own DNS servers, lets investigate

Cartoon of a detective investigate following footprints | Premium ...
ls -l /etc/resolv.conf

its a symlink to the another system file

/run/systemd/resolve/stub-resolv.conf

When you take a look at the directory where that file resides, there are two files

When you look at the other file you will see that /run/systemd/resolve/resolv.conf is the one which really is carrying the external name servers

You still can’t manually edit This file, and it gets updated by whatever the IPs provided as DNS servers via DHCP. netplan will dictate the IPs based on the static DNS servers you configure on Netplan YAML file

i can see there two entries, and they are the default Linode DNS servers discovered via DHCP, I’m going to keep them as is, since they are good enough for my use case

If you want to use your own servers here – Follow this guide

 Lets change the symlink to this file instead of the stub-resolve.conf

$ sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Now that its pointing to the right file

Lets restart the systemd-resolved

systemctl restart systemd-resolved

Now you can resolve DNS and install packages, etc

Docker compose script file for the PI-Hole

sudo mkdir /Docker_Images/
sudo mkdir /Docker_Images/Piholev6/

Lets navigate to this directory and start setting up our environment

nano /Docker_Images/Piholev6/docker-compose.yml
version: '3.4'
services:

   Pihole:
    container_name: pihole_v6
    image: pihole/pihole:latest
    hostname: Multicastbits-DNSService
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "8080:80/tcp"
      - "4343:443/tcp"
    environment:
      TZ: America/New_York
      DNS1: 1.1.1.1
      DNS2: 8.8.8.8
      WEBPASSWORD: F1ghtm4_Keng3n4sura
      ServerIP: 45.33.73.186
      enable_ipv6: "true"
      ServerIPv6: 2600:3c03::f03c:92ff:feb9:ea9c
    volumes:
       - '${ROOT}/pihole/etc-pihole/:/etc/pihole/'
       - '${ROOT}/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/'
    dns:
      - 127.0.0.1
      - 1.1.1.1
    cap_add:
      - NET_ADMIN
    restart: always

networks:
  default:
    external:
      name: Piholev6
networks:
  default:
    external:
      name: Piholev6

Lets break this down a littlebit

  • Version – Declare Docker compose version
  • container_name – This is the name of the container on the docker container registry
  • image – What image to pull from the Docker Hub
  • hostname – This is the host-name for the Docker container – this name will show up on your lookup when you are using this Pi-hole
  • ports – What ports should be NATed via the Docker Bridge to the host VM
  • TZ – Time Zone
  • DNS1 – DNS server used with in the image
  • DNS2 – DNS server used with in the image
  • WEBPASSWORD – Password for the Pi-Hole web console
  • ServerIP – Use the IPv4 address assigned to the VMs network interface(You need this for the Pi-Hole to respond on the IP for DNS queries)
  • IPv6 – Enable Disable IPv6 support
  • ServerIPv6 – Use the IPv4 address assigned to the VMs network interface (You need this for the Pi-Hole to respond on the IP for DNS queries)
  • volumes – These volumes will hold the configuration data so the container settings and historical data will persist reboots
  • cap_add:- NET_ADMIN – Add Linux capabilities to edit the network stack – link
  • restart: always – This will make sure the container gets restarted every time the VM boots up – Link
  • networks:default:external:name: Piholev6 – Set the container to use the network bridge we created before

Now lets bring up the Docker container

docker-compose up -d

-d switch will bring up the Docker container in the background

Run ‘Docker ps’ to confirm

Now you can access the web interface and use the Pihole

verifying its using the bridge network you created

Grab the network ID for the bridge network we create before and use the inspect switch to check the config

docker network ls
docker network inspect f7ba28db09ae

This will bring up the full configuration for the Linux bridge we created and the containers attached to the bridge will be visible under the “Containers”: tag

Testing

I manually configured my workstations primary DNS to the Pi-Hole IPs

Updating the docker Image

Pull the new image from the Registry

docker pull pihole/pihole

Take down the current container

docker-compose down

Run the new container

docker-compose up -d

Your settings will persist this update

Securing the install

now that we have a working Pi-Hole with IPv6 enabled, we can login and configure the Pihole server and resolve DNS as needed

but this is open to the public internet and will fall victim to DNS reflection attacks, etc

lets set up firewall rules and open up relevant ports (DNS, SSH, HTTPS) to the relevant IP addresses before we proceed

Disable IPtables from the docker daemon

Ubuntu uses UFW (uncomplicated firewall) as an obfuscation layer to make things easier for operators, but by default, Docker will open ports using IPtables with higher precedence, Rules added via UFW doesn’t take effect

So we need to tell docker not to do this when launching a container so we can manage the firewall rules via UFW

This file may not exist already if so nano will create it for you

sudo nano /etc/docker/daemon.json

Add the following lines to the file

{
"iptables": false
}

restart the docker services

sudo systemctl restart docker

now doing this might disrupt communication with the container until we allow them back in using UFW commands, so keep that in mind.

Automatically updating Firewall Rules based on the DYN DNS Host records

we are going to create a shell script and run it every hour using crontab

Shell Script Dry run

  • Get the IP from the DYNDNS Host records
  • remove/Cleanup existing rules
  • Add Default deny Rules
  • Add allow rules using the resolved IPs as the source

Dynamic IP addresses are updated on the following DNS records

  • trusted-Network01.selfip.net
  • trusted-Network02.selfip.net

Lets start by creating the script file under /bin/*

sudo touch /bin/PIHolefwruleupdate.sh
sudo chmod +x /bin/PIHolefwruleupdate.sh
sudo nano /bin/PIHolefwruleupdate.sh

now lets build the script

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
now=$(date +"%m/%d/%T")
DYNDNSNetwork01="trusted-Network01.selfip.net"
DYNDNSNetwork02="trusted-Network02.selfip.com"
#Get the network IP using dig
Network01_CurrentIP=`dig +short $DYNDNSNetwork01`
Network02_CurrentIP=`dig +short $DYNDNSNetwork02`
echo "-----------------------------------------------------------------"
echo Network A WAN IP $Network01_CurrentIP
echo Network B WAN IP $Network02_CurrentIP
echo "Script Run time : $now"
echo "-----------------------------------------------------------------"
#update firewall Rules
#reset firewall rules
#
sudo ufw --force reset
#
#Re-enable Firewall
#
sudo ufw --force enable
#
#Enable inbound default Deny firewall Rules
#
sudo ufw default deny incoming
#
#add allow Rules to the relevant networks
#
sudo ufw allow from $Network01_CurrentIP to any port 22 proto tcp
sudo ufw allow from $Network01_CurrentIP to any port 8080 proto tcp
sudo ufw allow from $Network01_CurrentIP to any port 53 proto udp
sudo ufw allow from $Network02_CurrentIP to any port 53 proto udp
#add the ipV6 DNS allow all Rule - Working on finding an effective way to lock this down, with IPv6 rick is minimal
sudo ufw allow 53/udp
#find and delete the allow any to any IPv4 Rule for port 53
sudo ufw --force delete $(ufw status numbered | grep '53*.*Anywhere.' | grep -v v6 | awk -F"[][]" '{print $2}')
echo "--------------------end Script------------------------------"

Lets run the script to make sure its working

I used a online port scanner to confirm

Setup Scheduled job with logging

lets use crontab and setup a scheduled job to run this script every hour

Make sure the script is copied to the /bin folder with the executable permissions

using crontab -e (If you are launching this for the first time it will ask you to pick the editor, I picked Nano)

crontab -e

Add the following line

0 * * * * /bin/PIHolefwruleupdate.sh >> /var/log/PIHolefwruleupdate_Cronoutput.log 2>&1
Lets break this down
0 * * * *

this will run the script every time minutes hit zero which is usually every hour

/bin/PIHolefwruleupdate.sh

Script Path to execute

/var/log/PIHolefwruleupdate_Cronoutput.log 2>&1

Log file with errors captured

Advertising VRF Connected/Static routes via MP BGP to OSPF – Guide Dell S4112F-ON – OS 10.5.1.3

Im going to base this off my VRF Setup and Route leaking article and continue building on top of it

Lets say we need to advertise connected routes within VRFs using IGP to an upstream or downstream iP address this is one of many ways to get to that objective

For this example we are going to use BGP to collect connected routes and advertise that over OSPF

Setup the BGP process to collect connected routes

router bgp 65000
 router-id 10.252.250.6
 !
 address-family ipv4 unicast
 !
 neighbor 10.252.250.1
!
vrf Tenant01_VRF
 !
 address-family ipv4 unicast
  redistribute connected
!
vrf Tenant02_VRF
 !
 address-family ipv4 unicast
  redistribute connected
!
vrf Tenant03_VRF
 !
 address-family ipv4 unicast
  redistribute connected
!
vrf Shared_VRF
 !
 address-family ipv4 unicast
  redistribute connected

Setup OSPF to Redistribute the routes collected via BGP

router ospf 250 vrf Shared_VRF
 area 0.0.0.0 default-cost 0
 redistribute bgp 65000
interface vlan250
 mode L3
 description OSPF_Routing
 no shutdown
 ip vrf forwarding Shared_VRF
 ip address 10.252.250.6/29
 ip ospf 250 area 0.0.0.0
 ip ospf mtu-ignore
 ip ospf priority 10

Testing and confirmation

Local OSPF Database

Remote device

VRF Setup with route leaking guide Dell S4112F-ON – OS 10.5.1.3

Scope –

Create Three VRFs for Three separate clients

Create a Shared VRF

Leak routes from each VRF to the Shared_VRF

Logical overview

Create the VRFs

ip vrf Tenant01_VRF
ip vrf Tenant02_VRF
ip vrf Tenant03_VRF

Create and initialize the Interfaces (SVI, Layer 3 interface, Loopback)

We are creating Layer 3 SVIs Per tenant

interface vlan200
 mode L3
 description Tenant01_NET01
 no shutdown
 ip vrf forwarding Tenant01_VRF
 ip address 10.251.100.254/24
!
interface vlan201
 mode L3
 description Tenant01_NET02
 no shutdown
 ip vrf forwarding Tenant01_VRF
 ip address 10.251.101.254/24
!
interface vlan210
 mode L3
 description Tenant02_NET01
 no shutdown
 ip vrf forwarding Tenant02_VRF
 ip address 172.17.100.254/24
!
interface vlan220
 no ip address
 description Tenant03_NET01
 no shutdown
 ip vrf forwarding Tenant03_VRF
 ip address 192.168.110.254/24
!
interface vlan250
 mode L3
 description OSPF_Routing
 no shutdown
 ip vrf forwarding Shared_VRF
 ip address 10.252.250.6/29

Confirmation

LABCORE# show i
image     interface inventory ip        ipv6      iscsi
LABCORE# show ip interface brief
Interface Name            IP-Address          OK       Method       Status     Protocol
=========================================================================================
Vlan 200                   10.251.100.254/24   YES      manual       up          up
Vlan 201                   10.251.101.254/24   YES      manual       up          up
Vlan 210                   172.17.100.254/24   YES      manual       up          up
Vlan 220                   192.168.110.254/24  YES      manual       up          up
Vlan 250                   10.252.250.6/29     YES      manual       up          up
LABCORE# show ip vrf
VRF-Name                          Interfaces

Shared_VRF                        Vlan250

Tenant01_VRF                      Vlan200-201

Tenant02_VRF                      Vlan210

Tenant03_VRF                      Vlan220

default                           Vlan1

management                        Mgmt1/1/1

Route leaking

For this Example we are going to Leak routes from each of these tenant VRFs in to the Shared VRF

This design will allow each VLAN within the VRFs to see each other, which can be a security issue how ever you can easily control this by

  • narrowing the routes down to hosts
  • Using Access-lists (not the most ideal but if you have a playbook you can program this in with out any issues)

Real world use cases may differ use this as a template on how to leak routes with in VRFs, update your config as needed

Create the route export statements wihtin the VRFS

ip vrf Shared_VRF
 ip route-import 2:100
 ip route-import 3:100
 ip route-import 4:100
 ip route-export 1:100
ip vrf Tenant01_VRF
 ip route-export 2:100
 ip route-import 1:100
ip vrf Tenant02_VRF
 ip route-export 3:100
 ip route-import 1:100
ip vrf Tenant03_VRF
 ip route-export 4:100
 ip route-import 1:100

Lets Explain this a bit

ip vrf Shared_VRF
 ip route-import 2:100 -----------> Import Leaked routes from target 2:100
 ip route-import 3:100 -----------> Import Leaked routes from target 3:100
 ip route-import 4:100 -----------> Import Leaked routes from target 4:100
 ip route-export 1:100  -----------> Export routes to target 1:100

if you need to filter out who can import the routes you need to use the route-map with prefixes to filter it out

Setup static routes per VRF as needed

ip route vrf Tenant01_VRF 10.251.100.0/24 interface vlan200
ip route vrf Tenant01_VRF 10.251.101.0/24 interface vlan201
!
ip route vrf Tenant02_VRF 172.17.100.0/24 interface vlan210
!
ip route vrf Tenant03_VRF 192.168.110.0/24 interface vlan220
!
ip route vrf Shared_VRF 0.0.0.0/0 10.252.250.1 interface vlan25
  • Now these static routes will be leaked and learned by the shared VRF
  • the Default route on the Shared VRF will be learned downstream by the tenant VRFs
  • instead of the default route on the shared VRF, if you scope it to a certain IP or a subnet you can prevent the traffic routing between the VRFs via the Shared VRF
  • if you need routes directly leaked between Tenents use the ip route-import on the VRF as needed

Confirmation

Routes are being distributed via internal BGP process

LABCORE# show ip route vrf Tenant01_VRF
Codes: C - connected
       S - static
       B - BGP, IN - internal BGP, EX - external BGP, EV - EVPN BGP
       O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
       E2 - OSPF external type 2, * - candidate default,
       + - summary route, > - non-active route
Gateway of last resort is via 10.252.250.1 to network 0.0.0.0
  Destination                 Gateway                                        Dist/Metric       Last Change
----------------------------------------------------------------------------------------------------------
  *B IN 0.0.0.0/0           via 10.252.250.1                                 200/0             12:17:42
  C     10.251.100.0/24     via 10.251.100.254       vlan200                 0/0               12:43:46
  C     10.251.101.0/24     via 10.251.101.254       vlan201                 0/0               12:43:46
LABCORE#
LABCORE# show ip route vrf Tenant02_VRF
Codes: C - connected
       S - static
       B - BGP, IN - internal BGP, EX - external BGP, EV - EVPN BGP
       O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
       E2 - OSPF external type 2, * - candidate default,
       + - summary route, > - non-active route
Gateway of last resort is via 10.252.250.1 to network 0.0.0.0
  Destination                 Gateway                                        Dist/Metric       Last Change
----------------------------------------------------------------------------------------------------------
  *B IN 0.0.0.0/0           via 10.252.250.1                                 200/0             12:17:45
  C     172.17.100.0/24     via 172.17.100.254       vlan210                 0/0               12:43:49
LABCORE#
LABCORE# show ip route vrf Tenant03_VRF
Codes: C - connected
       S - static
       B - BGP, IN - internal BGP, EX - external BGP, EV - EVPN BGP
       O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
       E2 - OSPF external type 2, * - candidate default,
       + - summary route, > - non-active route
Gateway of last resort is via 10.252.250.1 to network 0.0.0.0
  Destination                 Gateway                                        Dist/Metric       Last Change
----------------------------------------------------------------------------------------------------------
  *B IN 0.0.0.0/0           via 10.252.250.1                                 200/0             12:17:48
  C     192.168.110.0/24    via 192.168.110.254      vlan220                 0/0               12:43:52
LABCORE# show ip route vrf Shared_VRF
Codes: C - connected
       S - static
       B - BGP, IN - internal BGP, EX - external BGP, EV - EVPN BGP
       O - OSPF, IA - OSPF inter area, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type 2, E1 - OSPF external type 1,
       E2 - OSPF external type 2, * - candidate default,
       + - summary route, > - non-active route
Gateway of last resort is via 10.252.250.1 to network 0.0.0.0
  Destination                 Gateway                                        Dist/Metric       Last Change
----------------------------------------------------------------------------------------------------------
  *S    0.0.0.0/0           via 10.252.250.1         vlan250                 1/0               12:21:33
  B  IN 10.251.100.0/24     Direct,Tenant01_VRF      vlan200                 200/0             09:01:28
  B  IN 10.251.101.0/24     Direct,Tenant01_VRF      vlan201                 200/0             09:01:28
  C     10.252.250.0/29     via 10.252.250.6         vlan250                 0/0               12:42:53
  B  IN 172.17.100.0/24     Direct,Tenant02_VRF      vlan210                 200/0             09:01:28
  B  IN 192.168.110.0/24    Direct,Tenant03_VRF      vlan220                 200/0             09:02:09

We can ping outside to the internet from the VRF IPs

Redistribute leaked routes via IGP

You can use a Internal BGP process to pickup routes from any VRF and redistribute them to other IGP processes as needed – Check the Article for that information