Our 2-node Server 2012 Hyper-V cluster is having an issue where VMs seem to randomly fail for no apparent reason. We are using Dell R900s with a MD3200i SAN and we have separate networks for iSCSI and Heartbeat. On one of the nodes we get an
error 1069 "Cluster Resource 'Virtual Machine VM200X32' of type 'Virtual Machine' in clustered role 'VM200X32' failed." Below I have listed the relevant cluster log events from around the time it fails. The cluster has passed validation
and is running 50+ test VMs just fine, it's only a few of them that seem to be having this issue.
Just wondering if anyone else might have some input on what the problem could be.
Cluster event log:
2013/08/25-16:41:18.448 WARN [RHS] Resource Virtual Machine VM200X32 IsAlive has indicated failure.
2013/08/25-16:41:18.463 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'Virtual Machine VM200X32', gen(0) result 1/0.
2013/08/25-16:41:18.463 INFO [RCM] Res Virtual Machine VM200X32: Online -> ProcessingFailure( StateUnknown )
2013/08/25-16:41:18.463 INFO [RCM] TransitionToState(Virtual Machine VM200X32) Online-->ProcessingFailure.
2013/08/25-16:41:18.463 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (VM200X32, Online --> Pending)
2013/08/25-16:41:18.463 ERR [RCM] rcm::RcmResource::HandleFailure: (Virtual Machine VM200X32)
2013/08/25-16:41:18.463 INFO [RCM] resource Virtual Machine VM200X32: failure count: 0, restartAction: 0 persistentState: 1.
2013/08/25-16:41:18.463 INFO [RCM] Will queue immediate restart (500 milliseconds) of Virtual Machine VM200X32 after terminate is complete.
2013/08/25-16:41:18.463 INFO [RCM] Res Virtual Machine VM200X32: ProcessingFailure -> WaitingToTerminate( DelayRestartingResource )
2013/08/25-16:41:18.463 INFO [RCM] TransitionToState(Virtual Machine VM200X32) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
2013/08/25-16:41:18.463 INFO [RCM] Res Virtual Machine VM200X32: [WaitingToTerminate to DelayRestartingResource] -> Terminating( DelayRestartingResource )
2013/08/25-16:41:18.463 INFO [RCM] TransitionToState(Virtual Machine VM200X32) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
2013/08/25-16:41:18.463 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: Current state 'Online', event 'Terminate'
2013/08/25-16:41:18.463 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: State change 'Online' -> 'Terminated'
2013/08/25-16:41:18.463 INFO [RCM] ignored non-local state Pending for group VM200X32
2013/08/25-16:41:18.479 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine Configuration VM200X32', gen(0) result 0/0.
2013/08/25-16:41:18.479 INFO [RCM] Virtual Machine Configuration VM200X32: Flags 1 added to StatusInformation. New StatusInformation 1
2013/08/25-16:41:18.479 INFO [RCM] VM200X32: Added Flags 1 to StatusInformation. New StatusInformation 1
2013/08/25-16:41:18.479 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.275 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine Configuration VM200X32', gen(0) result 0/0.
2013/08/25-16:41:19.275 INFO [RCM] Virtual Machine Configuration VM200X32: Flags 1 removed from StatusInformation. New StatusInformation 0
2013/08/25-16:41:19.275 INFO [RCM] VM200X32: Removed Flags 1 from StatusInformation. New StatusInformation 0
2013/08/25-16:41:19.275 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.275 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: Current state 'Terminated', event 'VmStopped'
2013/08/25-16:41:19.306 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.836 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.836 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.836 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: State change 'Terminated' -> 'Offline'
2013/08/25-16:41:19.836 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.836 INFO [RCM] Res Virtual Machine VM200X32: [Terminating to DelayRestartingResource] -> DelayRestartingResource( StateUnknown )
2013/08/25-16:41:19.836 INFO [RCM] TransitionToState(Virtual Machine VM200X32) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
2013/08/25-16:41:19.836 WARN [RCM] Queueing immediate delay restart of resource Virtual Machine VM200X32 in 500 ms.
2013/08/25-16:41:20.351 INFO [RCM] Delay-restarting Virtual Machine VM200X32 and any waiting dependents.
2013/08/25-16:41:20.351 INFO [RCM-rbtr] giving default token to group VM200X32
2013/08/25-16:41:20.351 INFO [RCM-rbtr] giving default token to group VM200X32
2013/08/25-16:41:20.351 INFO [RCM] Res Virtual Machine VM200X32: DelayRestartingResource -> OnlineCallIssued( StateUnknown )
2013/08/25-16:41:20.351 INFO [RCM] TransitionToState(Virtual Machine VM200X32) DelayRestartingResource-->OnlineCallIssued.
2013/08/25-16:41:20.351 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: Current state 'Offline', event 'Online'
2013/08/25-16:41:20.351 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: State change 'Offline' -> 'OnlinePending'
2013/08/25-16:41:20.351 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'Virtual Machine VM200X32', gen(1) result 997/0.
2013/08/25-16:41:20.351 INFO [RCM] Res Virtual Machine VM200X32: OnlineCallIssued -> OnlinePending( StateUnknown )
2013/08/25-16:41:20.351 INFO [RCM] TransitionToState(Virtual Machine VM200X32) OnlineCallIssued-->OnlinePending.
2013/08/25-16:41:20.351 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:20.351 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine Configuration VM200X32', gen(0) result 0/0.
2013/08/25-16:41:20.351 INFO [RCM] Virtual Machine Configuration VM200X32: Flags 1 added to StatusInformation. New StatusInformation 1
2013/08/25-16:41:20.351 INFO [RCM] VM200X32: Added Flags 1 to StatusInformation. New StatusInformation 1
2013/08/25-16:41:20.367 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:20.694 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.911 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine Configuration VM200X32', gen(0) result 0/0.
2013/08/25-16:41:21.911 INFO [RCM] Virtual Machine Configuration VM200X32: Flags 1 removed from StatusInformation. New StatusInformation 0
2013/08/25-16:41:21.911 INFO [RCM] VM200X32: Removed Flags 1 from StatusInformation. New StatusInformation 0
2013/08/25-16:41:21.911 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.911 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: Current state 'OnlinePending', event 'VmRunning'
2013/08/25-16:41:21.942 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.942 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: 'Virtual Machine VM200X32' successfully started the virtual machine.
2013/08/25-16:41:21.958 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: State change 'OnlinePending' -> 'Online'
2013/08/25-16:41:21.958 INFO [RHS] Resource Virtual Machine VM200X32 has come online. RHS is about to report status change to RCM
2013/08/25-16:41:21.958 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.958 INFO [RCM] Res Virtual Machine VM200X32: OnlinePending -> Online( StateUnknown )
2013/08/25-16:41:21.958 INFO [RCM] TransitionToState(Virtual Machine VM200X32) OnlinePending-->Online.
2013/08/25-16:41:21.958 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (VM200X32, Pending --> Online)
2013/08/25-16:41:21.958 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.958 INFO [RCM] ignored non-local state Online for group VM200X32
Just wondering if anyone else might have some input on what the problem could be.
Cluster event log:
2013/08/25-16:41:18.448 WARN [RHS] Resource Virtual Machine VM200X32 IsAlive has indicated failure.
2013/08/25-16:41:18.463 INFO [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'Virtual Machine VM200X32', gen(0) result 1/0.
2013/08/25-16:41:18.463 INFO [RCM] Res Virtual Machine VM200X32: Online -> ProcessingFailure( StateUnknown )
2013/08/25-16:41:18.463 INFO [RCM] TransitionToState(Virtual Machine VM200X32) Online-->ProcessingFailure.
2013/08/25-16:41:18.463 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (VM200X32, Online --> Pending)
2013/08/25-16:41:18.463 ERR [RCM] rcm::RcmResource::HandleFailure: (Virtual Machine VM200X32)
2013/08/25-16:41:18.463 INFO [RCM] resource Virtual Machine VM200X32: failure count: 0, restartAction: 0 persistentState: 1.
2013/08/25-16:41:18.463 INFO [RCM] Will queue immediate restart (500 milliseconds) of Virtual Machine VM200X32 after terminate is complete.
2013/08/25-16:41:18.463 INFO [RCM] Res Virtual Machine VM200X32: ProcessingFailure -> WaitingToTerminate( DelayRestartingResource )
2013/08/25-16:41:18.463 INFO [RCM] TransitionToState(Virtual Machine VM200X32) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
2013/08/25-16:41:18.463 INFO [RCM] Res Virtual Machine VM200X32: [WaitingToTerminate to DelayRestartingResource] -> Terminating( DelayRestartingResource )
2013/08/25-16:41:18.463 INFO [RCM] TransitionToState(Virtual Machine VM200X32) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
2013/08/25-16:41:18.463 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: Current state 'Online', event 'Terminate'
2013/08/25-16:41:18.463 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: State change 'Online' -> 'Terminated'
2013/08/25-16:41:18.463 INFO [RCM] ignored non-local state Pending for group VM200X32
2013/08/25-16:41:18.479 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine Configuration VM200X32', gen(0) result 0/0.
2013/08/25-16:41:18.479 INFO [RCM] Virtual Machine Configuration VM200X32: Flags 1 added to StatusInformation. New StatusInformation 1
2013/08/25-16:41:18.479 INFO [RCM] VM200X32: Added Flags 1 to StatusInformation. New StatusInformation 1
2013/08/25-16:41:18.479 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.275 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine Configuration VM200X32', gen(0) result 0/0.
2013/08/25-16:41:19.275 INFO [RCM] Virtual Machine Configuration VM200X32: Flags 1 removed from StatusInformation. New StatusInformation 0
2013/08/25-16:41:19.275 INFO [RCM] VM200X32: Removed Flags 1 from StatusInformation. New StatusInformation 0
2013/08/25-16:41:19.275 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.275 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: Current state 'Terminated', event 'VmStopped'
2013/08/25-16:41:19.306 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.836 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.836 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.836 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: State change 'Terminated' -> 'Offline'
2013/08/25-16:41:19.836 INFO [RCM] HandleMonitorReply: TERMINATERESOURCE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:19.836 INFO [RCM] Res Virtual Machine VM200X32: [Terminating to DelayRestartingResource] -> DelayRestartingResource( StateUnknown )
2013/08/25-16:41:19.836 INFO [RCM] TransitionToState(Virtual Machine VM200X32) [Terminating to DelayRestartingResource]-->DelayRestartingResource.
2013/08/25-16:41:19.836 WARN [RCM] Queueing immediate delay restart of resource Virtual Machine VM200X32 in 500 ms.
2013/08/25-16:41:20.351 INFO [RCM] Delay-restarting Virtual Machine VM200X32 and any waiting dependents.
2013/08/25-16:41:20.351 INFO [RCM-rbtr] giving default token to group VM200X32
2013/08/25-16:41:20.351 INFO [RCM-rbtr] giving default token to group VM200X32
2013/08/25-16:41:20.351 INFO [RCM] Res Virtual Machine VM200X32: DelayRestartingResource -> OnlineCallIssued( StateUnknown )
2013/08/25-16:41:20.351 INFO [RCM] TransitionToState(Virtual Machine VM200X32) DelayRestartingResource-->OnlineCallIssued.
2013/08/25-16:41:20.351 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: Current state 'Offline', event 'Online'
2013/08/25-16:41:20.351 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: State change 'Offline' -> 'OnlinePending'
2013/08/25-16:41:20.351 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'Virtual Machine VM200X32', gen(1) result 997/0.
2013/08/25-16:41:20.351 INFO [RCM] Res Virtual Machine VM200X32: OnlineCallIssued -> OnlinePending( StateUnknown )
2013/08/25-16:41:20.351 INFO [RCM] TransitionToState(Virtual Machine VM200X32) OnlineCallIssued-->OnlinePending.
2013/08/25-16:41:20.351 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:20.351 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine Configuration VM200X32', gen(0) result 0/0.
2013/08/25-16:41:20.351 INFO [RCM] Virtual Machine Configuration VM200X32: Flags 1 added to StatusInformation. New StatusInformation 1
2013/08/25-16:41:20.351 INFO [RCM] VM200X32: Added Flags 1 to StatusInformation. New StatusInformation 1
2013/08/25-16:41:20.367 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:20.694 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.911 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine Configuration VM200X32', gen(0) result 0/0.
2013/08/25-16:41:21.911 INFO [RCM] Virtual Machine Configuration VM200X32: Flags 1 removed from StatusInformation. New StatusInformation 0
2013/08/25-16:41:21.911 INFO [RCM] VM200X32: Removed Flags 1 from StatusInformation. New StatusInformation 0
2013/08/25-16:41:21.911 INFO [RCM] HandleMonitorReply: LOCKEDMODE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.911 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: Current state 'OnlinePending', event 'VmRunning'
2013/08/25-16:41:21.942 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.942 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: 'Virtual Machine VM200X32' successfully started the virtual machine.
2013/08/25-16:41:21.958 INFO [RES] Virtual Machine <Virtual Machine VM200X32>: State change 'OnlinePending' -> 'Online'
2013/08/25-16:41:21.958 INFO [RHS] Resource Virtual Machine VM200X32 has come online. RHS is about to report status change to RCM
2013/08/25-16:41:21.958 INFO [RCM] HandleMonitorReply: ONLINERESOURCE for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.958 INFO [RCM] Res Virtual Machine VM200X32: OnlinePending -> Online( StateUnknown )
2013/08/25-16:41:21.958 INFO [RCM] TransitionToState(Virtual Machine VM200X32) OnlinePending-->Online.
2013/08/25-16:41:21.958 INFO [RCM] rcm::RcmGroup::UpdateStateIfChanged: (VM200X32, Pending --> Online)
2013/08/25-16:41:21.958 INFO [RCM] HandleMonitorReply: INMEMORY_NODELOCAL_PROPERTIES for 'Virtual Machine VM200X32', gen(1) result 0/0.
2013/08/25-16:41:21.958 INFO [RCM] ignored non-local state Online for group VM200X32