After looking around for a few hours digging in to the logs i figured out the issue, hopefully this helps some one else our there in the situation save some time.
Make sure the IPVS mode is enabled on the cluster configuration
If you are using :
RKE2 – edit the cluster.yaml file
RKE1 – Edit the cluster configuration from the rancher UI > Cluster management > Select the cluster > edit configuration > edit as YAML
Locate the services field under rancher_kubernetes_engine_config and add the following options to enable IPVS
Make sure the Kernel modules are enabled on the nodes running control planes
Background
Example Rancher – RKE1 cluster
sudo docker ps | grep proxy # find the container ID for kubproxy
sudo docker logs ####containerID###
0313 21:44:08.315888 108645 feature_gate.go:245] feature gates: &{map[]}
I0313 21:44:08.346872 108645 proxier.go:652] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="nf_conntrack_ipv4"
E0313 21:44:08.347024 108645 server_others.go:107] "Can't use the IPVS proxier" err="IPVS proxier will not be used because the following required kernel modules are not loaded: [ip_vs_lc]"
Kubproxy is trying to load the needed kernel modules and failing to enable IPVS
If you found this page you already know why you are looking for this, your server /dev/mapper/cs-root is filled due to /var/lib/docker taking up most of the space
Yes, you can change the location of the Docker overlay2 storage directory by modifying the daemon.json file. Here’s how to do it:
Open or create the daemon.json file using a text editor:
sudo nano /etc/docker/daemon.json
{
"data-root": "/path/to/new/location/docker"
}
Replace “/path/to/new/location/docker” with the path to the new location of the overlay2 directory.
If the file already contains other configuration settings, add the "data-root" setting to the file under the "storage-driver" setting:
Root CA cert is pushed out to all Servers/Desktops – This happens by default
Contents
Setup CA Certificate template
Deploy Auto-enrolled Certificates via Group Policy
Powershell logon script to set the WinRM listener
Deploy the script as a logon script via Group Policy
Testing
1 – Setup CA Certificate template to allow Client Servers/Desktops to checkout the certificate from the CA
Connect to the The Certification Authority Microsoft Management Console (MMC)
Navigate to Certificate Templates > Manage
On the “Certificate templates Console” window > Select Web Server > Duplicate Template
Under the new Template window Set the following attributes
General – Pick a Name and Validity Period – This is up to you
Compatibility – Set the compatibility attributes (You can leave this on the default values, It up to you)
Subject Name – Set ‘Subject Name’ attributes (Important)
Security – Add “Domain Computers” Security Group and Set the following permissions
Read – Allow
Enroll – Allow
Autoenroll – Allow
Click “OK” to save and close out of “Certificate template console”
Issue to the new template
Go back to the “The Certification Authority Microsoft Management Console” (MMC)
Under templates (Right click the empty space) > Select New > Certificate template to Issue
Under the Enable Certificate template window > Select the Template you just created
Allow few minutes for ADDS to replicate and pick up the changes with in the forest
2 – Deploy Auto-enrolled Certificates via Group Policy
Create a new GPO
Windows Settings > Security Settings > Public Key Policies/Certificate Services Client – Auto-Enrollment Settings
Link the GPO to the relevant OU with in your ADDS environment
Note – You can push out the root CA cert as a trusted root certificate with this same policy if you want to force computers to pick up the CA cert,
Testing
If you need to test it gpupdate/force or reboot your test machine, The Server VM/PC will pickup a certificate from ADCS PKI
3 – Powershell logon script to set the WINRM listener
Dry run
Setup the log file
Check for the Certificate matching the machines FQDN Auto-enrolled from AD CS
If exist
Set up the HTTPS WInRM listener and bind the certificate
Write log
else
Write log
#Malinda Rathnayake- 2020
#
#variable
$Date = Get-Date -Format "dd_MM_yy"
$port=5986
$SessionRunTime = Get-Date -Format "dd_yyyy_HH-mm"
#
#Setup Logs folder and log File
$ScriptVersion = '1.0'
$locallogPath = "C:\_Scripts\_Logs\WINRM_HTTPS_ListenerBinding"
#
$logging_Folder = (New-Item -Path $locallogPath -ItemType Directory -Name $Date -Force)
$ScriptSessionlogFile = New-Item $logging_Folder\ScriptSessionLog_$SessionRunTime.txt -Force
$ScriptSessionlogFilePath = $ScriptSessionlogFile.VersionInfo.FileName
#
#Check for the the auto-enrolled SSL Cert
$RootCA = "Company-Root-CA" #change This
$hostname = ([System.Net.Dns]::GetHostByName(($env:computerName))).Hostname
$certinfo = (Get-ChildItem -Path Cert:\LocalMachine\My\ |? {($_.Subject -Like "CN=$hostname") -and ($_.Issuer -Like "CN=$RootCA*")})
$certThumbprint = $certinfo.Thumbprint
#
#Script-------------------------------------------------------
#
#Remove the existing WInRM Listener if there is any
Get-ChildItem WSMan:\Localhost\Listener | Where -Property Keys -eq "Transport=HTTPS" | Remove-Item -Recurse -Force
#
#If the client certificate exists Setup the WinRM HTTPS listener with the cert else Write log
if ($certThumbprint){
#
New-Item -Path WSMan:\Localhost\Listener -Transport HTTPS -Address * -CertificateThumbprint $certThumbprint -HostName $hostname -Force
#
netsh advfirewall firewall add rule name="Windows Remote Management (HTTPS-In)" dir=in action=allow protocol=TCP localport=$port
#
Add-Content -Path $ScriptSessionlogFilePath -Value "Certbinding with the HTTPS WinRM HTTPS Listener Completed"
Add-Content -Path $ScriptSessionlogFilePath -Value "$certinfo.Subject"}
else{
Add-Content -Path $ScriptSessionlogFilePath -Value "No Cert matching the Server FQDN found, Please run gpupdate/force or reboot the system"
}
Script is commented with Explaining each section (should have done functions but i was pressed for time, never got around to do it, if you do fix it up and improve this please let me know in the comments :D)
5 – Deploy the script as a logon script via Group Policy
Setup a GPO and set this script as a logon Powershell script
Im using a user policy with GPO Loop-back processing set to Merge applied to the server OU
Testing
To confirm WinRM is listening on HTTPS, type the following commands:
winrm enumerate winrm/config/listener
Winrm get http://schemas.microsoft.com/wbem/wsman/1/config
Let’s talk a little bit about this code and unpack this
Vagrant API version
Vagrant uses API versions for its configuration file, this is how it can stay backward compatible. So in every Vagrantfile we need to specify which version to use. The current one is version 2 which works with Vagrant 1.1 and up.
Provisioning the Ansible VM
This will
Provision the controller Ubuntu VM
Create a bridged network adapter
Set the host-name – LAB-Controller
Set the static IP – 172.17.10.120/24
Run the Shell script that installs Ansible using apt-get install (We will get to this below)
Lets start digging in…
Specifying the Controller VM Name, base box and hostname
Vagrant uses a base image to clone a virtual machine quickly. These base images are known as “boxes” in Vagrant, and specifying the box to use for your Vagrant environment is always the first step after creating a new Vagrantfile.
Define the shell script to customize the VM config and install the Ansible Package
Now this is where we define the provisioning shell script
this script installs Ansible and set the host file entries to make your life easier
In case you are wondering VLS stands for V=virtual,L – linux S – server.
I use this naming scheme for my VMs. Feel free to use anything you want; make sure it matches what you defined on the Vagrantfile under node.vm.hostname
We covered most of the code used above, the only difference here is we are using each method to create 3 VMs with the same template (I’m lazy and it’s more convenient)
This will create three Ubuntu VMs with the following Host-names and IP addresses, you should update these values to match you LAN, or use a private Adapter
vls-node1 – 172.17.10.121
vls-node2 – 172.17.10.122
vls-node1 – 172.17.10.123
So now that we are done with explaining the code, let’s run this
Building the Lab environment using Vagrant
Issue the following command to check your syntax
Vagrant status
Issue the following command to bring up the environment
Vagrant up
If you get this message Reboot in to UEFI and make sure virtualization is enabled
Intel – VT-D
AMD Ryzen – SVM
If everything is kumbaya you will see vagrant firing up the deployment
It will provision 4 VMs as we specified
Notice since we have the “vagrant-vbguest” plugin installed, it will reinstall the relevant guest tools along with the dependencies for the OS
==> vls-node3: Machine booted and ready!
[vls-node3] No Virtualbox Guest Additions installation found.
rmmod: ERROR: Module vboxsf is not currently loaded
rmmod: ERROR: Module vboxguest is not currently loaded
Reading package lists...
Building dependency tree...
Reading state information...
Package 'virtualbox-guest-x11' is not installed, so not removed
The following packages will be REMOVED:
virtualbox-guest-utils*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 5799 kB disk space will be freed.
(Reading database ... 61617 files and directories currently installed.)
Removing virtualbox-guest-utils (6.0.14-dfsg-1) ...
Processing triggers for man-db (2.8.7-3) ...
(Reading database ... 61604 files and directories currently installed.)
Purging configuration files for virtualbox-guest-utils (6.0.14-dfsg-1) ...
Processing triggers for systemd (242-7ubuntu3.7) ...
Reading package lists...
Building dependency tree...
Reading state information...
linux-headers-5.3.0-51-generic is already the newest version (5.3.0-51.44).
linux-headers-5.3.0-51-generic set to manually installed.
Check the status
Vagrant status
Testing
Connecting via SSH to your VMs
vagrant ssh controller
“Controller” is the VMname we defined before not the hostname, You can find this by running Vagrant status on posh or your terminal
We are going to connect to our controller and check everything
Little bit more information on the networking side
Vagrant Adds two interfaces, for each VM
NIC 1 – Nat’d in to the host (control plane for Vagrant to manage the VMs)
NIC 2 – Bridged adapter we provisioned in the script with the IP Address
Default route is set via the Private(NAT’d) interface (you cant change it)
Netplan configs
Vagrant creates a custom netplan yaml for interface configs
I hope this helped someone. when I started with Vagrant a few years back it took me a few tries to figure out the system and the logic behind it, this will give you a basic understanding on how things are plugged together.
let me know in the comments if you see any issues or mistakes.
Received the following error from the Azure AD stating that Password Synchronization was not working on the tenant.
When i manually initiate a delta sync, i see the following logs
"The Specified Domain either does not exist or could not be contacted"
(click to enlarge)
Checked the following
Restarted ADsync Services
Resolve the ADDS Domain FQDN and DNS – Working
Test required ports for AD-sync using portqry – issues with the Primary ADDS server defined on the DNS values
Root Cause
Turns out the Domain controller Defined as the primary DNS value was pointing was going thorough updates, its responding on the DNS but doesn’t return any data (Brown-out state)
Assumption
when checking DNS since the DNS server is connecting, Windows doesn’t check the secondary and tertiary servers defined under DNS servers.
This might happen if you are using a ADDS server via a S2S tunnel/MPLS when the latency goes high
Resolution
Check make sure your ADDS-DNS servers defined on AD-SYNC server are alive and responding
in my case i just updated the “Primary” DNS value with the umbrella Appliance IP (this act as a proxy and handle the fail-over)
So recently I ran into this annoying error message with Exchange 2016 CU11 Update.
Environment info-
Exchange 2016 upgrade from CU8 to CU11
Exchange binaries are installed under D:\Microsoft\Exchange_Server_V15\..
Microsoft.PowerShell.Commands.GetItemCommand.ProcessRecord()". [12/04/2018 16:41:43.0233] [1] [ERROR] Cannot find path 'D:\Microsoft\Exchange_Server_V15\UnifiedMessaging\grammars' because it does not exist. [12/04/2018 16:41:43.0233] [1] [ERROR-REFERENCE] Id=UnifiedMessagingComponent___99d8be02cb8d413eafc6ff15e437e13d Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup [12/04/2018 16:41:43.0234] [1] Setup is stopping now because of one or more critical errors. [12/04/2018 16:41:43.0234] [1] Finished executing component tasks. [12/04/2018 16:41:43.0318] [1] Ending processing Install-UnifiedMessagingRole [12/04/2018 16:44:51.0116] [0] CurrentResult setupbase.maincore:396: 0 [12/04/2018 16:44:51.0118] [0] End of Setup [12/04/2018 16:44:51.0118] [0] **********************************************
Root Cause
Ran the Setup again and it failed with the same error while going though the log files i notice that the setup looks for this file path while configuring the "Mailbox role: Unified Messaging service" (Stage 6 on the GUI installer)
There was no folder present with the name grammarsunder the Path specified on the error
just to confirm, i checked another server on CU8 and the grammars folder is there.
Not sure why the folder got removed, it may have happened during the first run of the CU11 setup that failed,
Resolution
My first thought was to copy the folder from an existing CU8 server. but just to avoid any issues (since exchange is sensitive to file versions) I created an empty folder with the name "grammars" under D:\Microsoft\Exchange_Server_V15\UnifiedMessaging\
Ran the setup again and it continued the upgrade process and completed without any issues...¯\_(ツ)_/¯
[12/04/2018 18:07:50.0416] [2] Ending processing Set-ServerComponentState [12/04/2018 18:07:50.0417] [2] Beginning processing Write-ExchangeSetupLog [12/04/2018 18:07:50.0420] [2] Install is complete. Server state has been set to Active. [12/04/2018 18:07:50.0421] [2] Ending processing Write-ExchangeSetupLog [12/04/2018 18:07:50.0422] [1] Finished executing component tasks. [12/04/2018 18:07:50.0429] [1] Ending processing Start-PostSetup [12/04/2018 18:07:50.0524] [0] CurrentResult setupbase.maincore:396: 0 [12/04/2018 18:07:50.0525] [0] End of Setup [12/04/2018 18:07:50.0525] [0] **********************************************
Considering cost of this software M$ really have to be better about error handling IMO, i have run in to silly issues like this way too many times since Exchange 2010.
For part 1 of this series we are going to cover the following
Dual Stack Setup
DHCPV6 configuration and explanation
– Guide –
I used my a Netgate router running PfSense to terminate the 6in4 tunnel.it adds the firewall and monitoring capabilities on your Ipv6 network
Before we begin, we need to make a few adjustments on the firewall
Allow IPv6 Traffic
On new installations of pfSense after 2.1, IPv6 traffic is allowed by default. If the configuration on the firewall has been upgraded from older versions, then IPv6 would still be blocked. To enable IPv6 traffic on PFsense, perform the following:
Navigate to System > Advanced on the Networking tab
Check Allow IPv6 if not already checked
Click Save
Allow ICMP
ICMP echo requests must be allowed on the WAN address that is terminating the tunnel to ensure that it is online and reachable.
Firewall> Rules > WAN
Create a regular tunnel.
Enter your IPv4 address as the tunnel’s endpoint address.
Note – After entering your IPv4 address, the website will check to make sure that it can ping your machine. If it cannot ping your machine, you will get an error like the one below:
You can access the tunnel information from the accounts page
While you are here go to “Advance Tab” and setup an “Update key”. (We need it later)
Create and Assign the GIF Interface
Next, create the interface for the GIF tunnel in pfSense. Complete the fields with the corresponding information from the tunnel broker configuration summary.
Navigate to Interfaces > (assign) on the GIF tab.
Click Add to add a new entry.
Set the Parent Interface to the WAN where the tunnel terminates. This would be the WAN which has the Client IPv4 Address on the tunnel broker.
Set the GIF Remote Address in pfSense to the Server IPv4 Address on the summary.
Set the GIF Tunnel Local Address in pfSense to the Client IPv6 Address on the summary.
Set the GIF Tunnel Remote Address in pfSense to the Server IPv6 Address on the summary, along the with prefix length (typically / 64).
Leave remaining options blank or unchecked.
Enter a Description.
Click Save.
Example GIF Tunnel.
Assign GIF Interface
Click on Interfaces > (Assignments)
choose the GIF interface to be used for an OPT interface. In this example, the OPT interface has been renamed WAN_HP_NET_IPv6. Click Save and Apply Changes if they appear.
Configure OPT Interface
With the OPT interface assigned, Click on the OPT interface from the Interfaces menu to enable it Keep IPv6 Configuration Type set to None.
Setup the IPv6 Gateway
When the interface is configured as listed above, a dynamic IPv6 gateway is added automatically, but it is not yet marked as default.
Navigate to System > Routing
Edit the dynamic IPv6 gateway with the same name as the IPv6 WAN created above.
Check Default Gateway.
Click Save.
Click Apply Changes.
Status > Gateways to view the gateway status. The gateway will show as “Online” if the configuration is successful
Set Up the LAN Interface for IPv6
The LAN interface may be configured for static IPv6 network. The network used for IPv6 addressing on the LAN Interface is an address in the Routed /64 or /48 subnet assigned by the tunnel broker.
The Routed /64 or /48 is the basis for the IPv6 Address field
For this exercise we are going to use ::1 for the LAN interface IP from the Prefixes provided above
Routed /64 : 2001:470:1f07:79a::/64
Interface IP – 2001:470:1f07:79a::1
Set Up DHCPv6 and RA (Router Advertisements)
Now that we have the tunnel up and running we need to make sure devices behind the lan interface can get a IPv6 address
There are couple of ways to handle the addressing
Sateless Auto Address Configuration (SLAAC)
SLAAC just means Stateless Auto Address Configuration, but it shouldn’t be confused with Stateless DHCPv6. In fact, we are talking about two different approaches.
SLAAC is the simplest way to give an IPv6 address to a client, because it exclusively rely on Neighbor Discovery Protocol. This protocol, that we simply call NDP, allows devices on a network to discover their Layer 3 neighbors. We use it to retrieve the layer 2 reachability information, like ARP, and to find out routers on the network.
When a device comes online, it sends a Router Solicitation message. It’s basically asking “Are there some routers out there?”. If we have a router on the same network, that router will reply with a Router Advertisement(RA) message. Using this message, the router will tell the client some information about the network:
Who is the default gateway (the link-local address of the router itself)
What is the global unicast prefix (for example, 2001:DB8:ACAD:10::/64)
With these information, the client is going to create a new global unicast address using the EUI-64 technique. Now the client has an IP address from the global unicast prefix range of the router, and that address is valid over the Internet.
This method is extremely simple, and requires virtually no configuration. However, we can’t centralize it and we cannot specify further information, such as DNS settings. To do that, we need to use a DHCPv6 technique
Just like IP v4 we need to setup DHCP for the IPv6 range for the devices behind the firewall to use SLAAT
Stateless DHCPv6
Stateless DHCPv6 brings to the picture the DHCPv6 protocol. With this approach, we still use SLAAC to obtain reachability information, and we use DHCPv6 for extra items.
The client always starts with a Router Solicitation, and the router on the segment responds with a Router Advertisement. This time, the Router Advertisement has a flag called other-config set to 1. Once the client receives the message, it will still use SLAAC to craft its own IPv6 address. However, the flag tells the client to do something more.
After the SLAAC process succeed, the client will craft a DHCPv6 request and send it through the network. A DHCPv6 server will eventually reply with all the extra information we needed, such as DNS server or domain name.
This approach is called stateless since the DHCPv6 server does not manage any lease for the clients. Instead, it just gives extra information as needed.
Configuring IPv6 Router Advertisements
Router Advertisements (RA) tell an IPv6 network not only which routers are available to reach other networks, but also tell clients how to obtain an IPv6 address. These options are configured per-interface and work similar to and/or in conjunction with DHCPv6.
DHCPv6 is not able to send clients a router for use as a gateway as is traditionally done with IPv4 DHCP. The task of announcing gateways falls to RA.
Operating Mode: Controls how clients behave. All modes advertise this firewall as a router for IPv6. The following modes are available:
Router Only: Clients will need to set addresses statically
Unmanaged: Client addresses obtained only via Stateless Address Autoconfiguration (SLAAC).
Managed: Client addresses assigned only via DHCPv6.
Assisted: Client addresses assigned by either DHCPv6 or SLAAC (or both).
Enable DHCPv6 Server on the interface
Setup IPv6 DNS Addresses
we are going to use cloud-flare DNS (At the time of writing CF is rated as the fastest resolver by Thousandeyes.com)
dnsomatic.com is wonderful free service to update your dynamic IP on multiple locations, i used this because if needed i have the freedom to change routers/firewalls with out messing up my config (Im using a one of my RasPi’s to update DNS-O-Matic)
im working on another article for this, will link it to this section ASAP
Few Notes –
Android OS, Chrome OS still doesn’t support DHCPv6
Mac OSX and windows 10, Server 2016 uses and prefers Ipv6
Check the windows firewall rules if you have issues with NAT rules and manually update rules
Your MTU will drop-down since you are sending the IPv6 headers encapsulated in the Ipv4 packets.Personally i have no issues with my Ipv6 network Behind a spectrum DOCSIS modem. but this may cause issues depending on your ISP ie : CGNat