i have 2 esxi servers,on each one i have a virtual domain controller
fist domain controller is dc1:192.168.20.2/24
second one is dc2 : 192.168.20.10/24
domain controllers can ping together and every thing was ok till 12 weeks ago, but now servers cant replicate and i cant create object on neither
when i want create an object( ex new user) it give me this error:windows cannot create object. the directiry service was unable to alocate a relative identifire
also when i want to replicate from dc1 to dc2 it give access denied error
the result of dc diag on them is :
dc diag on dc1 server
Domain Controller Diagnosis
Performing initial setup:
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\dc1
Starting test: Connectivity
......................... dc1 passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\dc1
Starting test: Replications
[Replications Check,dc1] DsReplicaGetInfoW(PENDING_OPS) failed w
ith error 8453,
Win32 Error 8453.
......................... dc1 failed test Replications
Starting test: NCSecDesc
......................... dc1 passed test NCSecDesc
Starting test: NetLogons
User credentials does not have permission to perform this operation.
The account used for this test must have network logon privileges
for the target machine's domain.
......................... dc1 failed test NetLogons
Starting test: Advertising
......................... dc1 passed test Advertising
Starting test: KnowsOfRoleHolders
......................... dc1 passed test KnowsOfRoleHolders
Starting test: RidManager
......................... dc1 passed test RidManager
Starting test: MachineAccount
......................... dc1 passed test MachineAccount
Starting test: Services
......................... dc1 passed test Services
Starting test: ObjectsReplicated
......................... dc1 passed test ObjectsReplicated
Starting test: frssysvol
......................... dc1 passed test frssysvol
Starting test: frsevent
There are warning or error events within the last 24 hours after the
SYSVOL has been shared. Failing SYSVOL replication problems may cause
Group Policy problems.
......................... dc1 failed test frsevent
Starting test: kccevent
......................... dc1 passed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0xC0001B59
Time Generated: 12/23/2012 10:10:32
(Event String could not be retrieved)
An Error Event occured. EventID: 0xC0001B59
Time Generated: 12/23/2012 10:28:26
(Event String could not be retrieved)
An Error Event occured. EventID: 0xC0001B59
Time Generated: 12/23/2012 10:41:42
(Event String could not be retrieved)
An Error Event occured. EventID: 0xC0001B59
Time Generated: 12/23/2012 10:57:43
(Event String could not be retrieved)
......................... dc1 failed test systemlog
Starting test: VerifyReferences
......................... dc1 passed test VerifyReferences
Running partition tests on : ForestDnsZones
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom
Running partition tests on : DomainDnsZones
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom
Running partition tests on : Schema
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom
Running partition tests on : Configuration
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom
Running partition tests on : SP
Starting test: CrossRefValidation
......................... SP passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... SP passed test CheckSDRefDom
Running enterprise tests on : SP.Local
Starting test: Intersite
......................... SP.Local passed test Intersite
Starting test: FsmoCheck
......................... SP.Local passed test FsmoCheck
__________________________________________________________________________________________________________
dcdiaG ON ADDITIONAL
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = dc2
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\dc2
Starting test: Connectivity
......................... dc2 passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\dc2
Starting test: Advertising
......................... dc2 passed test Advertising
Starting test: FrsEvent
There are warning or error events within the last 24 hours after the
SYSVOL has been shared. Failing SYSVOL replication problems may cause
Group Policy problems.
......................... dc2 passed test FrsEvent
Starting test: DFSREvent
......................... dc2 passed test DFSREvent
Starting test: SysVolCheck
......................... dc2 passed test SysVolCheck
Starting test: KccEvent
......................... dc2 passed test KccEvent
Starting test: KnowsOfRoleHolders
......................... dc2 passed test KnowsOfRoleHolders
Starting test: MachineAccount
......................... dc2 passed test MachineAccount
Starting test: NCSecDesc
......................... dc2 passed test NCSecDesc
Starting test: NetLogons
[dc2] User credentials does not have permission to perform this
operation.
The account used for this test must have network logon privileges
for this machine's domain.
......................... dc2 failed test NetLogons
Starting test: ObjectsReplicated
......................... dc2 passed test ObjectsReplicated
Starting test: Replications
[Replications Check,dc2] A recent replication attempt failed:
From dc1 to dc2
Naming Context: DC=ForestDnsZones,DC=SP,DC=Local
The replication generated an error (1256):
The remote system is not available. For information about network tr
oubleshooting, see Windows Help.
The failure occurred at 2012-12-23 10:15:16.
The last success occurred at 2012-12-03 14:26:08.
445 failures have occurred since the last success.
[Replications Check,dc2] A recent replication attempt failed:
From dc1 to dc2
Naming Context: DC=DomainDnsZones,DC=SP,DC=Local
The replication generated an error (5):
Access is denied.
The failure occurred at 2012-12-23 10:30:04.
The last success occurred at 2012-12-03 15:06:45.
697 failures have occurred since the last success.
[Replications Check,dc2] A recent replication attempt failed:
From dc1 to dc2
Naming Context: CN=Schema,CN=Configuration,DC=SP,DC=Local
The replication generated an error (5):
Access is denied.
The failure occurred at 2012-12-23 10:15:16.
The last success occurred at 2012-12-03 14:26:08.
435 failures have occurred since the last success.
[Replications Check,dc2] A recent replication attempt failed:
From dc1 to dc2
Naming Context: CN=Configuration,DC=SP,DC=Local
The replication generated an error (5):
Access is denied.
The failure occurred at 2012-12-23 10:15:16.
The last success occurred at 2012-12-03 14:26:08.
438 failures have occurred since the last success.
REPLICATION LATENCY WARNING
ERROR: Expected notification link is missing.
Source dc1
Replication of new changes along this path will be delayed.
This problem should self-correct on the next periodic sync.
[Replications Check,dc2] A recent replication attempt failed:
From dc1 to dc2
Naming Context: DC=SP,DC=Local
The replication generated an error (5):
Access is denied.
The failure occurred at 2012-12-23 10:55:54.
The last success occurred at 2012-12-03 15:06:48.
11334 failures have occurred since the last success.
......................... dc2 failed test Replications
Starting test: RidManager
The DS has corrupt data: rIDPreviousAllocationPool value is not valid
No rids allocated -- please check eventlog.
......................... dc2 failed test RidManager
Starting test: Services
Could not open NTDS Service on dc2, error 0x5
"Access is denied."
......................... dc2 failed test Services
Starting test: SystemLog
A warning event occurred. EventID: 0x8000001D
Time Generated: 12/23/2012 10:15:10
Event String:
The Key Distribution Center (KDC) cannot find a suitable certificate
to use for smart card logons, or the KDC certificate could not be verified. Sma
rt card logon may not function correctly if this problem is not resolved. To cor
rect this problem, either verify the existing KDC certificate using certutil.exe
or enroll for a new KDC certificate.
......................... dc2 passed test SystemLog
Starting test: VerifyReferences
......................... dc2 passed test VerifyReferences
Running partition tests on : ForestDnsZones
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ForestDnsZones passed test
CrossRefValidation
Running partition tests on : DomainDnsZones
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... DomainDnsZones passed test
CrossRefValidation
Running partition tests on : Schema
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Running partition tests on : Configuration
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Running partition tests on : SP
Starting test: CheckSDRefDom
......................... SP passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... SP passed test CrossRefValidation
Running enterprise tests on : SP.Local
Starting test: LocatorCheck
......................... SP.Local passed test LocatorCheck
Starting test: Intersite
......................... SP.Local passed test Intersite