Network/telecom engineer here, barely fluent in AD.
Our customer's environment features a domain with 20+ sites. My team is integrating three mobile datacenters that can be moved around on a truck, similar to those used in police, disaster recovery, or energy exploration applications. Each mobile
datacenter, which I'll call MD, includes a domain controller (Server 2016), an Exchange Server 2016, and an array of application servers, and end-user devices.
The MDs are connected to the parent network via microwave or SATCOM links. Since link availability is often subject to terrain and other factors, AD and Exchange will be required to function within the site when there is no link available. This
means when the link is down, MD users must still be able to log into clients, send emails to other users, and utilize NTFS-based file sharing. When the link to the network is re-established, AD should replicate, and Exchange should send any queued messages.
Presently, each MD is in its own child domain. Everything is working correctly when the link to the parent domain is up. When the link goes down, all MD site assets are still able to communicate with each other - as expected.
The problem scenario is when the servers in the MCs are booted up from a cold start, as they would be after transport, AD presents the following of error and warning events:
<domain controller>
5719
Error
NETLOGON
System
10/7/2019 7:14:41 PM
<domain controller>
1129
Error
Microsoft-Windows-GroupPolicy
System
10/7/2019 7:14:12 PM
<domain controller>
10154
Warning
Microsoft-Windows-Windows Remote Management
System
10/7/2019 7:14:10 PM
<domain controller>
7039
Warning
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:13:59 PM
<domain controller>
1014
Warning
Microsoft-Windows-DNS Client Events
System
10/7/2019 7:13:42 PM
<domain controller>
1014
Warning
Microsoft-Windows-DNS Client Events
System
10/7/2019 7:13:42 PM
<domain controller>
10016
Error
Microsoft-Windows-DistributedCOM
System
10/7/2019 7:12:15 PM
<domain controller>
1008
Error
Microsoft-Windows-Perflib
Application
10/7/2019 3:32:24 PM
<domain controller>
5
Error
Microsoft-Windows-Security-Kerberos
System
10/7/2019 1:02:36 PM
DCDIAG.exe output is below:
C:\Users\user>dcdiag
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = <server>
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: <domain\server>
Starting test: Connectivity
An error that is usually temporary occurred during DNS host lookup. Please try again later.
Got error while checking LDAP and RPC connectivity. Please check your firewall settings.
......................... <server> failed test Connectivity
Doing primary tests
Testing server: <domain\server>
Skipping all tests, because server <server> is not responding to directory service requests.
Running partition tests on : DomainDnsZones
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation
Running partition tests on : <domain>
Starting test: CheckSDRefDom
......................... <domain> passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... <domain> passed test CrossRefValidation
Running partition tests on : ForestDnsZones
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation
Running partition tests on : Schema
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Running partition tests on : Configuration
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Running enterprise tests on : <domain>
Starting test: LocatorCheck
......................... <domain> passed test LocatorCheck
Starting test: Intersite
......................... <domain> passed test Intersite
The Exchange server has 14 services that will not start, and also has a list of warning/error events:
<Exchange Server>
1015
Error
MSExchangeDiagnostics
Application
10/7/2019 7:23:30 PM
<Exchange Server>
10010
Error
Microsoft-Windows-DistributedCOM
System
10/7/2019 7:22:23 PM
<Exchange Server>
2142
Error
MSExchangeADTopology
Application
10/7/2019 7:22:22 PM
<Exchange Server>
4999
Error
MSExchange Common
Application
10/7/2019 7:22:12 PM
<Exchange Server>
4999
Error
MSExchange Common
Application
10/7/2019 7:21:57 PM
<Exchange Server>
4999
Error
MSExchange Common
Application
10/7/2019 7:21:57 PM
<Exchange Server>
1693
Warning
MSExchange Unified Messaging
Application
10/7/2019 7:21:57 PM
<Exchange Server>
4999
Error
MSExchange Common
Application
10/7/2019 7:21:57 PM
<Exchange Server>
4999
Error
MSExchange Common
Application
10/7/2019 7:21:57 PM
<Exchange Server>
1031
Error
MSExchangeSubmission
Application
10/7/2019 7:21:57 PM
<Exchange Server>
4999
Error
MSExchange Common
Application
10/7/2019 7:21:57 PM
<Exchange Server>
4999
Error
MSExchange Common
Application
10/7/2019 7:21:57 PM
<Exchange Server>
4096
Error
MSExchangeRepl
Application
10/7/2019 7:21:39 PM
<Exchange Server>
2155
Error
MSExchangeRepl
Application
10/7/2019 7:21:39 PM
<Exchange Server>
7023
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:20:59 PM
<Exchange Server>
1023
Error
MSExchangeIS
Application
10/7/2019 7:20:59 PM
<Exchange Server>
7005
Error
MSExchangeTransportLogSearch
Application
10/7/2019 7:20:59 PM
<Exchange Server>
2001
Error
Filtering ADConnector
Application
10/7/2019 7:20:59 PM
<Exchange Server>
10007
Error
MSExchange Mid-Tier Storage
Application
10/7/2019 7:20:59 PM
<Exchange Server>
16019
Warning
MSExchangeTransportDelivery
Application
10/7/2019 7:20:59 PM
<Exchange Server>
16019
Warning
MSExchangeTransportSearch
Application
10/7/2019 7:20:59 PM
<Exchange Server>
16019
Warning
MSExchangeFrontEndTransport
Application
10/7/2019 7:20:59 PM
<Exchange Server>
1045
Warning
MSExchange EdgeSync
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
1070
Error
MSExchange EdgeSync
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4127
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
2031
Error
Filtering ADConnector
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
2060
Error
Filtering ADConnector
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
2142
Error
MSExchangeADTopology
Application
10/7/2019 7:20:59 PM
<Exchange Server>
2120
Error
MSExchange ADAccess
Application
10/7/2019 7:20:59 PM
<Exchange Server>
6027
Error
Microsoft-Filtering-FIPFS
Application
10/7/2019 7:20:23 PM
<Exchange Server>
10010
Error
Microsoft-Windows-DistributedCOM
System
10/7/2019 7:20:23 PM
<Exchange Server>
2104
Error
MSExchangeADTopology
Application
10/7/2019 7:19:59 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:19:22 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:19:22 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:19:22 PM
<Exchange Server>
4027
Error
MSExchange ADAccess
Application
10/7/2019 7:19:22 PM
<Exchange Server>
2501
Error
MSExchange ADAccess
Application
10/7/2019 7:18:58 PM
<Exchange Server>
4999
Error
MSExchange Common
Application
10/7/2019 7:18:57 PM
<Exchange Server>
1028
Error
MSExchangeRPC
Application
10/7/2019 7:18:57 PM
<Exchange Server>
24
Warning
Microsoft-Windows-Time-Service
System
10/7/2019 7:18:30 PM
<Exchange Server>
10010
Error
Microsoft-Windows-DistributedCOM
System
10/7/2019 7:18:23 PM
<Exchange Server>
1015
Error
MSExchangeDiagnostics
Application
10/7/2019 7:18:18 PM
<Exchange Server>
10154
Warning
Microsoft-Windows-Windows Remote Management
System
10/7/2019 7:18:08 PM
<Exchange Server>
1032
Error
MSExchangeDiagnostics
Application
10/7/2019 7:18:06 PM
<Exchange Server>
1015
Error
MSExchangeDiagnostics
Application
10/7/2019 7:18:06 PM
<Exchange Server>
1015
Error
MSExchangeDiagnostics
Application
10/7/2019 7:18:06 PM
<Exchange Server>
1012
Error
MSExchangeDiagnostics
Application
10/7/2019 7:17:43 PM
<Exchange Server>
1002
Error
MSExchangeThrottling
Application
10/7/2019 7:17:27 PM
<Exchange Server>
7000
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:17:26 PM
<Exchange Server>
7009
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:17:26 PM
<Exchange Server>
2142
Error
MSExchangeADTopology
Application
10/7/2019 7:17:22 PM
<Exchange Server>
7010
Warning
MSExchangeFrontEndTransport
Application
10/7/2019 7:17:13 PM
<Exchange Server>
7010
Warning
MSExchangeTransportDelivery
Application
10/7/2019 7:17:13 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
7022
Error
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:16:56 PM
<Exchange Server>
10010
Error
Microsoft-Windows-DistributedCOM
System
10/7/2019 7:16:23 PM
<Exchange Server>
134
Warning
Microsoft-Windows-Time-Service
System
10/7/2019 7:14:46 PM
<Exchange Server>
134
Warning
Microsoft-Windows-Time-Service
System
10/7/2019 7:14:46 PM
<Exchange Server>
134
Warning
Microsoft-Windows-Time-Service
System
10/7/2019 7:14:42 PM
<Exchange Server>
1014
Warning
Microsoft-Windows-DNS Client Events
System
10/7/2019 7:14:41 PM
<Exchange Server>
8016
Warning
Microsoft-Windows-DNS Client Events
System
10/7/2019 7:14:34 PM
<Exchange Server>
1129
Error
Microsoft-Windows-GroupPolicy
System
10/7/2019 7:14:33 PM
<Exchange Server>
129
Warning
Microsoft-Windows-Time-Service
System
10/7/2019 7:14:33 PM
<Exchange Server>
5719
Error
NETLOGON
System
10/7/2019 7:14:33 PM
<Exchange Server>
1014
Warning
Microsoft-Windows-DNS Client Events
System
10/7/2019 7:14:32 PM
<Exchange Server>
7039
Warning
Microsoft-Windows-Service Control Manager
System
10/7/2019 7:14:21 PM
<Exchange Server>
10149
Warning
Microsoft-Windows-Windows Remote Management
System
10/7/2019 7:13:01 PM
<Exchange Server>
6003
Error
MSExchange SACL Watcher
Application
10/7/2019 7:13:00 PM
<Exchange Server>
10016
Error
Microsoft-Windows-DistributedCOM
System
10/7/2019 7:13:00 PM
<Exchange Server>
5719
Error
NETLOGON
System
10/7/2019 7:12:51 PM
<Exchange Server>
1015
Error
MSExchangeDiagnostics
Application
10/7/2019 7:12:06 PM
When I bring the link back up, within seconds the problems resolve themselves on both servers and they begin running normally.
Is there any way to cause AD/Exchange in a child domain to start up normally with no connection to parent network?