Azure AD Sync Connect No-Start-Connection status


Received the following error from the Azure AD stating that Password Synchronization was not working on the tenant.

When i manually initiate a delta sync, i see the following logs

"The Specified Domain either does not exist or could not be contacted"

(click to enlarge)

Checked the following

  • Restarted ADsync Services
  • Resolve the ADDS Domain FQDN and DNS – Working
  • Test required ports for AD-sync using portqry – issues with the Primary ADDS server defined on the DNS values

Root Cause

Turns out the Domain controller Defined as the primary DNS value was pointing was going thorough updates, its responding on the DNS but doesn’t return any data (Brown-out state)


when checking DNS since the DNS server is connecting, Windows doesn’t check the secondary and tertiary servers defined under DNS servers.

This might happen if you are using a ADDS server via a S2S tunnel/MPLS when the latency goes high


Check make sure your ADDS-DNS servers defined on AD-SYNC server are alive and responding

in my case i just updated the “Primary” DNS value with the umbrella Appliance IP (this act as a proxy and handle the fail-over)

Upgrading VMware EXSI Hosts using Vcenter Update Manager Baseline (6.5 to 6.7 Update 2)

Update Manager is bundled in the vCenter Server Appliance since version 6.5, it’s a plug-in that runs on the vSphere Web Client.  we can use the component to

  • patch/upgrade hosts
  • deploy .vib files within the V-Center
  • Scan your VC environment and report on any out of compliance hosts

Hardcore/Experienced VMware operators will scoff at this article, but I have seen many organizations still using ILO/IDRAC to mount an ISO to update hosts and they have no idea this function even exists.

Now that’s out of the way Let’s get to the how-to part of this

In Vcenter click the “Menu” and drill down to the “Update Manager”

This Blade will show you all the nerd knobs and overview of your current Updates and compliance levels

Click on the “Baselines” Tab

You will have two predefined baselines for security patches created by the Vcenter, let keep that aside for now

Navigate to the “ESXi Images” Tab, and Click “Import”

Once the Upload is complete, Click on “New Baseline”

Fill in the Name and Description that makes sense to anyone that logs in and click Next

Select the image you just Uploaded before on the next Screen and continue through the wizard and complete it

Note – If you have other 3rd party software for ESXI you can create seprate baselines for those and use baseline Groups to push out upgrades and vib files at the same time 

Now click the “Menu” and Navigate Backup to “Hosts and Clusters”

Now you can apply the Baseline this at various levels within the Vcenter Hierarchy

Vcenter | DataCenter | Cluster | Host

Depending on your use case pick the right level

Excerpt from the KB 

For ESXi hosts in a cluster, the remediation process is sequential by default. With Update Manager, you can select to run host remediation in parallel.

When you remediate a cluster of hosts sequentially and one of the hosts fails to enter maintenance mode, Update Manager reports an error, and the process stops and fails. The hosts in the cluster that are remediated stay at the updated level. The ones that are not remediated after the failed host remediation are not updated. If a host in a DRS enabled cluster runs a virtual machine on which Update Manager or vCenter Server are installed, DRS first attempts to migrate the virtual machine running vCenter Server or Update Manager to another host so that the remediation succeeds. In case the virtual machine cannot be migrated to another host, the remediation fails for the host, but the process does not stop. Update Manager proceeds to remediate the next host in the cluster.

The host upgrade remediation of ESXi hosts in a cluster proceeds only if all hosts in the cluster can be upgraded.

Remediation of hosts in a cluster requires that you temporarily disable cluster features such as VMware DPM and HA admission control. Also, turn off FT if it is enabled on any of the virtual machines on a host, and disconnect the removable devices connected to the virtual machines on a host, so that they can be migrated with vMotion. Before you start a remediation process, you can generate a report that shows which cluster, host, or virtual machine has the cluster features enabled.

Link to KB on Remediation

Moving on; for this example, since I have only 2 hosts. we are going apply the baseline at the cluster level but apply the remediation at host level

Host 1 > Enter Maintenance > Remediation > Update complete and online

Host 2 > Enter Maintenance > Remediation > Update complete and online

Select the cluster, Click the “Updates” Tab and click on “Attach” on the Attached baselines section

Select and attach the baseline we created before

Click “Check Compliance” to scan and get a report

Select the host in the cluster, enter maintenance mode

Click “REMEDIATE” to start the upgrade. (if you do this at a cluster level if you have DRS, Update Manager will update each node)

This will reboot the host and go through the update process

Foot Notes –

You might run into the following issue

“vCenter cannot deploy Host upgrade agent to host”

Cause 1

Scratch partition is full use Vcenter and change the scratch folder location


Creating a persistent scratch location for ESXi  –

Cause 2

Hardware is not compatible,

I had this issue due to 6.7 dropping support for an LSI Raid card on an older firmware, you need to do some foot work and check the log files to figure out why its failing

Vmware HCL – Link

ESXI and Vcenter log file locations – link

MS Exchange 2016 [ERROR] Cannot find path ‘..\Exchange_Server_V15\UnifiedMessaging\grammars’ because it does not exist.

So recently I ran into this annoying error message with Exchange 2016 CU11 Update.

Environment info-

  • Exchange 2016 upgrade from CU8 to CU11
  • Exchange binaries are installed under D:\Microsoft\Exchange_Server_V15\..
Microsoft.PowerShell.Commands.GetItemCommand.ProcessRecord()". [12/04/2018 16:41:43.0233] [1] [ERROR] Cannot find path 'D:\Microsoft\Exchange_Server_V15\UnifiedMessaging\grammars' because it does not exist. 
[12/04/2018 16:41:43.0233] [1] [ERROR-REFERENCE] Id=UnifiedMessagingComponent___99d8be02cb8d413eafc6ff15e437e13d Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup
[12/04/2018 16:41:43.0234] [1] Setup is stopping now because of one or more critical errors. [12/04/2018 16:41:43.0234] [1] Finished executing component tasks.
[12/04/2018 16:41:43.0318] [1] Ending processing Install-UnifiedMessagingRole
[12/04/2018 16:44:51.0116] [0] CurrentResult setupbase.maincore:396: 0 [12/04/2018 16:44:51.0118] [0] End of Setup
[12/04/2018 16:44:51.0118] [0] **********************************************

Root Cause

Ran the Setup again and it failed with the same error
while going though the log files i notice that the setup looks for this file path while configuring the "Mailbox role: Unified Messaging service" (Stage 6 on the GUI installer)

$grammarPath = join-path $RoleInstallPath "UnifiedMessaging\grammars\*";

There was no folder present with the name grammars under the Path specified on the error

just to confirm, i checked another server on CU8 and the grammars folder is there.

Not sure why the folder got removed, it may have happened during the first run of the CU11 setup that failed,


My first thought was to copy the folder from an existing CU8 server. but just to avoid any issues (since exchange is sensitive to file versions)
I created an empty folder with the name "grammars" under D:\Microsoft\Exchange_Server_V15\UnifiedMessaging\

Ran the setup again and it continued the upgrade process and completed without any issues...¯\_(ツ)_/¯

[12/04/2018 18:07:50.0416] [2] Ending processing Set-ServerComponentState
[12/04/2018 18:07:50.0417] [2] Beginning processing Write-ExchangeSetupLog
[12/04/2018 18:07:50.0420] [2] Install is complete. Server state has been set to Active.
[12/04/2018 18:07:50.0421] [2] Ending processing Write-ExchangeSetupLog
[12/04/2018 18:07:50.0422] [1] Finished executing component tasks.
[12/04/2018 18:07:50.0429] [1] Ending processing Start-PostSetup
[12/04/2018 18:07:50.0524] [0] CurrentResult setupbase.maincore:396: 0
[12/04/2018 18:07:50.0525] [0] End of Setup
[12/04/2018 18:07:50.0525] [0] **********************************************

Considering cost of this software M$ really have to be better about error handling IMO, i have run in to silly issues like this way too many times since Exchange 2010.

IP version 6 with Dual-stack using a Tunnel broker 6in4 – PFSense/ASA -Part 01

If your ISP doesn’t have Native IP version 6 Support with Dual Stack  here is a workaround to get it setup for your home lab enviroment

What you need

> Router/Firewall/UTM that supports IPv6 Tunneling

  • PFsense/OpenSense/VyOS
  • DD-WRT 
  • Cisco ISR
  • Juniper SRX

> Active Account with an Ipv6 Tunnel Broker

      For this example we are going to be using Hurricane Electric Free IPv6 Tunnel Broker

Overview of the setup

For part 1 of this series  we are going to cover the following

  • Dual Stack Setup
  • DHCPV6 configuration and explanation

– Guide –

I used my a Netgate router running PfSense to terminate the 6in4 adds the firewall and monitoring capabilities on your Ipv6 network

Before we begin, we need to make a few adjustments on the firewall

Allow IPv6 Traffic

On new installations of pfSense after 2.1, IPv6 traffic is allowed by default. If the configuration on the firewall has been upgraded from older versions, then IPv6 would still be blocked. To enable IPv6 traffic on PFsense, perform the following:

  • Navigate to System > Advanced on the Networking tab
  • Check Allow IPv6 if not already checked
  • Click Save

Allow ICMP

ICMP echo requests must be allowed on the WAN address that is terminating the tunnel to ensure that it is online and reachable.

Firewall> Rules > WAN
Create a regular tunnel.

Enter your IPv4 address as the tunnel’s endpoint address.

Note – After entering your IPv4 address, the website will check to make sure that it can ping your machine. If it cannot ping your machine, you will get an error like the one below:

You can access the tunnel information from the accounts page

While you are here go to “Advance Tab” and setup an “Update key”. (We need it later)

Create and Assign the GIF Interface

Next, create the interface for the GIF tunnel in pfSense. Complete the fields with the corresponding information from the tunnel broker configuration summary.

  • Navigate to Interfaces > (assign) on the GIF tab.
  • Click fa-plus Add to add a new entry.
  • Set the Parent Interface to the WAN where the tunnel terminates. This would be the WAN which has the Client IPv4 Address on the tunnel broker.
  • Set the GIF Remote Address in pfSense to the Server IPv4 Address on the summary.
  • Set the GIF Tunnel Local Address in pfSense to the Client IPv6 Address on the summary.
  • Set the GIF Tunnel Remote Address in pfSense to the Server IPv6 Address on the summary, along the with prefix length (typically / 64).
  • Leave remaining options blank or unchecked.
  • Enter a Description.
  • Click Save.

Example GIF Tunnel.

Assign GIF Interface

Click fa-plus on Interfaces > (Assignments)

choose the GIF interface to be used for an OPT interface. In this example, the OPT interface has been renamed WAN_HP_NET_IPv6. Click Save and Apply Changes if they appear.


Configure OPT Interface

With the OPT interface assigned, Click on the OPT interface from the Interfaces menu to enable it  Keep IPv6 Configuration Type set to None.

Setup the IPv6 Gateway

When the interface is configured as listed above, a dynamic IPv6 gateway is added automatically, but it is not yet marked as default.

  • Navigate to System > Routing
  • Edit the dynamic IPv6 gateway with the same name as the IPv6 WAN created above.
  • Check Default Gateway.
  • Click Save.
  • Click Apply Changes.
Status > Gateways to view the gateway status. The gateway will show as “Online” if the configuration is successful

Set Up the LAN Interface for IPv6

The LAN interface may be configured for static IPv6 network. The network used for IPv6 addressing on the LAN Interface is an address in the Routed /64 or /48 subnet assigned by the tunnel broker.

  • The Routed /64 or /48 is the basis for the IPv6 Address field

For this exercise we are going to use ::1 for the LAN interface IP from the Prefixes provided above

Routed /64 : 2001:470:1f07:79a::/64

Interface IP – 2001:470:1f07:79a::1

Set Up DHCPv6 and RA (Router Advertisements)

Now that we have the tunnel up and running we need to make sure devices behind the lan interface can get a IPv6 address

There are couple of ways to handle the addressing

Sateless Auto Address Configuration (SLAAC)

SLAAC just means Stateless Auto Address Configuration, but it shouldn’t be confused with Stateless DHCPv6. In fact, we are talking about two different approaches.

SLAAC is the simplest way to give an IPv6 address to a client, because it exclusively rely on Neighbor Discovery Protocol. This protocol, that we simply call NDP, allows devices on a network to discover their Layer 3 neighbors. We use it to retrieve the layer 2 reachability information, like ARP, and to find out routers on the network.

When a device comes online, it sends a Router Solicitation message. It’s basically asking “Are there some routers out there?”. If we have a router on the same network, that router will reply with a Router Advertisement (RA) message. Using this message, the router will tell the client some information about the network:

  • Who is the default gateway (the link-local address of the router itself)
  • What is the global unicast prefix (for example, 2001:DB8:ACAD:10::/64)

With these information, the client is going to create a new global unicast address using the EUI-64 technique. Now the client has an IP address from the global unicast prefix range of the router, and that address is valid over the Internet.

This method is extremely simple, and requires virtually no configuration. However, we can’t centralize it and we cannot specify further information, such as DNS settings. To do that, we need to use a DHCPv6 technique

Just like IP v4 we need to setup DHCP for the IPv6 range for the devices behind the firewall to use SLAAT

Stateless DHCPv6

Stateless DHCPv6 brings to the picture the DHCPv6 protocol. With this approach, we still use SLAAC to obtain reachability information, and we use DHCPv6 for extra items.

The client always starts with a Router Solicitation, and the router on the segment responds with a Router Advertisement. This time, the Router Advertisement has a flag called other-config set to 1. Once the client receives the message, it will still use SLAAC to craft its own IPv6 address. However, the flag tells the client to do something more.

After the SLAAC process succeed, the client will craft a DHCPv6 request and send it through the network. A DHCPv6 server will eventually reply with all the extra information we needed, such as DNS server or domain name.

This approach is called stateless since the DHCPv6 server does not manage any lease for the clients. Instead, it just gives extra information as needed.

Configuring IPv6 Router Advertisements

Router Advertisements (RA) tell an IPv6 network not only which routers are available to reach other networks, but also tell clients how to obtain an IPv6 address. These options are configured per-interface and work similar to and/or in conjunction with DHCPv6.

DHCPv6 is not able to send clients a router for use as a gateway as is traditionally done with IPv4 DHCP. The task of announcing gateways falls to RA.

Operating Mode: Controls how clients behave. All modes advertise this firewall as a router for IPv6. The following modes are available:

  • Router Only: Clients will need to set addresses statically
  • Unmanaged: Client addresses obtained only via Stateless Address Autoconfiguration (SLAAC).
  • Managed: Client addresses assigned only via DHCPv6.
  • Assisted: Client addresses assigned by either DHCPv6 or SLAAC (or both).

Enable DHCPv6 Server on the interface

Setup IPv6 DNS Addresses

we are going to use cloud-flare DNS (At the time of writing CF is rated as the fastest resolver by

  • 2606:4700:4700::1111
  • 2606:4700:4700::1001

Keeping your Tunnel endpoint Address Updated with your Dynamic IP

This only applies if you have a dynamic IPv4 from your ISP

As you may remember from our first step when registering the 6in4 tunnel on the website we had to enter our Public IP and enable ICMP

We need to make sure we keep this updated when our IP changes ovetime

There are few ways to accomplish this

  • Use PFsense DynDNS feature  is wonderful free service to update your dynamic IP on multiple locations, i used this because if needed i have the freedom to change routers/firewalls with out messing up my config (Im using a one of my RasPi’s to update DNS-O-Matic)

im working on another article for this, will link it to this section ASAP


Few Notes –

Android OS, Chrome OS still doesn’t support DHCPv6

Mac OSX and windows 10, Server 2016 uses and prefers Ipv6

Check the windows firewall rules if you have issues with NAT rules and manually update rules

Your MTU will drop-down since you are sending the IPv6 headers encapsulated in the Ipv4 packets.Personally i have no issues with my Ipv6 network Behind a spectrum DOCSIS modem. but this may cause issues depending on your ISP ie : CGNat

Here is a good write up


Part 2

With Part two of this series we will use an ASA for IPv6 using the PFsense router as an tunnel-endpoint

Example Network

Link spotlight

– Understanding IPv6 EUI-64 Bit Address

– IPv6 Stateless Auto Configuration

– Configure the ASA to Pass IPv6 Traffic

– Setup IPv6 TunnelBroker – NetGate

– ipv6-at-home Part 1 | Part II | Part III

Until next time….