Hello,
First thing this morning, I started experiencing issues with a few of my non PDC AD servers. Running DCDiag revealed a number of errors on the secondary DCs.
Here is my dcdiag output on the AD3 DC:
Doing initial required tests
Testing server: Site\AD3
Starting test: Connectivity
......................... AD3 passed test Connectivity
Doing primary tests
Testing server: Site\AD3
Starting test: Advertising
......................... AD3 passed test Advertising
Starting test: FrsEvent
......................... AD3 passed test FrsEvent
Starting test: DFSREvent
There are warning or error events within the last 24 hours after the SYSVOL has been shared. Failing SYSVOL
replication problems may cause Group Policy problems.
......................... AD3 failed test DFSREvent
Starting test: SysVolCheck
......................... AD3 passed test SysVolCheck
Starting test: KccEvent
A warning event occurred. EventID: 0x8000061E
Time Generated: 03/18/2019 11:42:10
Event String:
All directory servers in the following site that can replicate the directory partition over this transport are currently unavailable.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.
A warning event occurred. EventID: 0x8000061E
Time Generated: 03/18/2019 11:42:10
Event String:
All directory servers in the following site that can replicate the directory partition over this transport are currently unavailable.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.
A warning event occurred. EventID: 0x8000061E
Time Generated: 03/18/2019 11:42:10
Event String:
All directory servers in the following site that can replicate the directory partition over this transport are currently unavailable.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.
A warning event occurred. EventID: 0x8000061E
Time Generated: 03/18/2019 11:42:10
Event String:
All directory servers in the following site that can replicate the directory partition over this transport are currently unavailable.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:42:10
Event String:
The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local site.
A warning event occurred. EventID: 0x80000785
Time Generated: 03/18/2019 11:42:10
Event String:
The attempt to establish a replication link for the following writable directory partition failed.
A warning event occurred. EventID: 0x80000785
Time Generated: 03/18/2019 11:42:10
Event String:
The attempt to establish a replication link for the following writable directory partition failed.
A warning event occurred. EventID: 0x80000785
Time Generated: 03/18/2019 11:42:10
Event String:
The attempt to establish a replication link for the following writable directory partition failed.
A warning event occurred. EventID: 0x80000785
Time Generated: 03/18/2019 11:42:10
Event String:
The attempt to establish a replication link for the following writable directory partition failed.
A warning event occurred. EventID: 0x80000785
Time Generated: 03/18/2019 11:42:10
Event String:
The attempt to establish a replication link for the following writable directory partition failed.
......................... AD3 failed test KccEvent
Starting test: KnowsOfRoleHolders
[AD1] DsBindWithSpnEx() failed with error -2146893022,
The target principal name is incorrect..
Warning: AD1 is the Schema Owner, but is not responding to DS RPC Bind.
[AD1] LDAP bind failed with error 8341,
A directory service error has occurred..
Warning: AD1 is the Schema Owner, but is not responding to LDAP Bind.
Warning: AD1 is the Domain Owner, but is not responding to DS RPC Bind.
Warning: AD1 is the Domain Owner, but is not responding to LDAP Bind.
Warning: AD1 is the PDC Owner, but is not responding to DS RPC Bind.
Warning: AD1 is the PDC Owner, but is not responding to LDAP Bind.
Warning: AD1 is the Rid Owner, but is not responding to DS RPC Bind.
Warning: AD1 is the Rid Owner, but is not responding to LDAP Bind.
Warning: AD1 is the Infrastructure Update Owner, but is not responding to DS RPC Bind.
Warning: AD1 is the Infrastructure Update Owner, but is not responding to LDAP Bind.
......................... AD3 failed test KnowsOfRoleHolders
Starting test: MachineAccount
......................... AD3 passed test MachineAccount
Starting test: NCSecDesc
......................... AD3 passed test NCSecDesc
Starting test: NetLogons
......................... AD3 passed test NetLogons
Starting test: ObjectsReplicated
......................... AD3 passed test ObjectsReplicated
Starting test: Replications
REPLICATION-RECEIVED LATENCY WARNING
AD3: Current time is 2019-03-18 11:42:45.
DC=ForestDnsZones,DC=ad,DC=domain,DC=com
Last replication received from AD2 at
2019-02-27 21:37:33
Last replication received from AD1 at
2019-02-27 21:38:13
DC=DomainDnsZones,DC=ad,DC=domain,DC=com
Last replication received from AD2 at
2019-02-27 21:37:33
Last replication received from AD1 at
2019-02-27 21:38:21
CN=Schema,CN=Configuration,DC=ad,DC=domain,DC=com
Last replication received from AD2 at
2019-02-27 21:37:33
Last replication received from AD1 at
2019-02-27 21:38:13
CN=Configuration,DC=ad,DC=domain,DC=com
Last replication received from AD2 at
2019-02-27 21:37:33
Last replication received from AD1 at
2019-02-27 21:38:13
DC=ad,DC=domain,DC=com
Last replication received from AD2 at
2019-02-27 21:37:33
Last replication received from AD1 at
2019-02-27 21:42:28
......................... AD3 passed test Replications
Starting test: RidManager
......................... AD3 failed test RidManager
Starting test: Services
......................... AD3 passed test Services
Starting test: SystemLog
An error event occurred. EventID: 0x40000004
Time Generated: 03/18/2019 11:08:39
Event String:
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server ad1$. The target name used was ldap/AD1.ad.domain.com. This indicates that the target server failed to decrypt the ticket
provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN is only registered on the account used by the server. This error can
also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server and the KDC are both configured to use the same password. If the
server name is not fully qualified, and the target domain (AD.DOMAIN.COM) is different from the client domain (AD.DOMAIN.COM), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.
An error event occurred. EventID: 0x40000004
Time Generated: 03/18/2019 11:12:10
Event String:
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server ad1$. The target name used was E3514235-4B06-11D1-AB04-00C04FC2DCD2/98e808a5-c419-48fa-b5b1-c64f03eb83df/ad.domain.com@ad.domain.com.
This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN
is only registered on the account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server
and the KDC are both configured to use the same password. If the server name is not fully qualified, and the target domain (AD.DOMAIN.COM) is different from the client domain (AD.DOMAIN.COM), check if there are identically named server accounts in these two
domains, or use the fully-qualified name to identify the server.
An error event occurred. EventID: 0x40000004
Time Generated: 03/18/2019 11:22:57
Event String:
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server ad1$. The target name used was E3514235-4B06-11D1-AB04-00C04FC2DCD2/98E808A5-C419-48FA-B5B1-C64F03EB83DF/ad.domain.com@ad.domain.com.
This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN
is only registered on the account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server
and the KDC are both configured to use the same password. If the server name is not fully qualified, and the target domain (AD.DOMAIN.COM) is different from the client domain (AD.DOMAIN.COM), check if there are identically named server accounts in these two
domains, or use the fully-qualified name to identify the server.
An error event occurred. EventID: 0x40000004
Time Generated: 03/18/2019 11:27:10
Event String:
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server ad1$. The target name used was LDAP/98e808a5-c419-48fa-b5b1-c64f03eb83df._msdcs.ad.domain.com. This indicates that the
target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN is only registered on the
account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server and the KDC are both configured
to use the same password. If the server name is not fully qualified, and the target domain (AD.DOMAIN.COM) is different from the client domain (AD.DOMAIN.COM), check if there are identically named server accounts in these two domains, or use the fully-qualified
name to identify the server.
......................... AD3 failed test SystemLog
Starting test: VerifyReferences
......................... AD3 passed test VerifyReferences
Running partition tests on : ForestDnsZones
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation
Running partition tests on : DomainDnsZones
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation
Running partition tests on : Schema
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Running partition tests on : Configuration
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Running partition tests on : ad
Starting test: CheckSDRefDom
......................... ad passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ad passed test CrossRefValidation
Running enterprise tests on : ad.domain.com
Starting test: LocatorCheck
......................... ad.domain.com passed test LocatorCheck
Starting test: Intersite
......................... ad.domain.com passed test Intersite
So I checked on the PDC and found the following:
Doing initial required tests
Testing server: Site\AD1
Starting test: Connectivity
......................... AD1 passed test Connectivity
Doing primary tests
Testing server: Site\AD1
Starting test: Advertising
......................... AD1 passed test Advertising
Starting test: FrsEvent
......................... AD1 passed test FrsEvent
Starting test: DFSREvent
......................... AD1 passed test DFSREvent
Starting test: SysVolCheck
......................... AD1 passed test SysVolCheck
Starting test: KccEvent
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:45:07
Event String: The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:45:07
Event String: The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local
site.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:45:07
Event String: The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:45:07
Event String: The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local
site.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:45:07
Event String: The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:45:07
Event String: The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local
site.
An error event occurred. EventID: 0xC000051F
Time Generated: 03/18/2019 11:45:07
Event String: The Knowledge Consistency Checker (KCC) has detected problems with the following directory partition.
A warning event occurred. EventID: 0x80000749
Time Generated: 03/18/2019 11:45:07
Event String: The Knowledge Consistency Checker (KCC) was unable to form a complete spanning tree network topology. As a result, the following list of sites cannot be reached from the local
site.
......................... AD1 failed test KccEvent
Starting test: KnowsOfRoleHolders
......................... AD1 passed test KnowsOfRoleHolders
Starting test: MachineAccount
......................... AD1 passed test MachineAccount
Starting test: NCSecDesc
......................... AD1 passed test NCSecDesc
Starting test: NetLogons
......................... AD1 passed test NetLogons
Starting test: ObjectsReplicated
......................... AD1 passed test ObjectsReplicated
Starting test: Replications
REPLICATION-RECEIVED LATENCY WARNING
AD1: Current time is 2019-03-18 11:50:47.
CN=Schema,CN=Configuration,DC=ad,DC=domain,DC=com
Last replication received from AD4 at
2019-02-27 15:00:11
CN=Configuration,DC=ad,DC=domain,DC=com
Last replication received from AD4 at
2019-02-27 15:00:11
DC=ad,DC=domain,DC=com
Last replication received from AD4 at
2019-02-27 15:00:12
......................... AD1 passed test Replications
Starting test: RidManager
......................... AD1 passed test RidManager
Starting test: Services
......................... AD1 passed test Services
Starting test: SystemLog
......................... AD1 passed test SystemLog
Starting test: VerifyReferences
......................... AD1 passed test VerifyReferences
Running partition tests on : ForestDnsZones
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation
Running partition tests on : DomainDnsZones
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation
Running partition tests on : Schema
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Running partition tests on : Configuration
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Running partition tests on : ad
Starting test: CheckSDRefDom
......................... ad passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ad passed test CrossRefValidation
Running enterprise tests on : ad.domain.com
Starting test: LocatorCheck
......................... ad.domain.com passed test LocatorCheck
Starting test: Intersite
......................... ad.domain.com passed test Intersite
If I go into sites and services, and manually force the sync between AD3 and AD1, I get the following:
The following error occurred during the attempt to synchronize naming context CN=Configuration,DC=ad,DC=domain,DC=com from Domain Controller AD1 to Domain Controller AD3: The target principal name is incorrect.
The operation will not continue.
I've looked to see if there are duplicate SPNs on the PDC (AD1) but I don't see any duplicates.
The other odd thing is the result I get when I run the following:
C:\Windows\system32>netdom verify ad3
The secure channel from AD3 to the domain DOMAIN has been verified. The connection
is with the machine \\AD1.AD.DOMAIN.COM.
I'm not sure what broke. I haven't changed any admin passwords recently. I'm stumped. Any ideas or suggestions?
Thanks!