oh yes, forgot to increase logging level.
already changed managed IP Address.
increased to verbose.
at time of disconnection before and when disconnects
vmkernel.log
2013-01-17T14:22:19.462Z cpu7:72726)VmMemMigrate: vm 72726: 5005: Regular swap file bitmap checks out.
2013-01-17T14:22:19.464Z cpu7:72726)VMotion: 5458: 1358432531942417 D: Resume handshake successful
2013-01-17T14:22:19.531Z cpu6:72891)Hbr: 3405: Migration end received (worldID=72726) (migrateType=1) (event=1) (isSource=0) (sharedConfig=1)
2013-01-17T14:22:19.551Z cpu1:72892)Swap: vm 72726: 3254: Starting prefault for the migration swap file
2013-01-17T14:22:19.563Z cpu1:72892)Swap: vm 72726: 3429: Finish swapping in migration swap file. (faulted 0 pages, pshared 0 pages). Success.
2013-01-17T14:22:19.713Z cpu2:7925)Config: 347: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)
2013-01-17T14:22:23.851Z cpu0:4104)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124007d23c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-01-17T14:22:23.851Z cpu0:4104)ScsiDeviceIO: 2316: Cmd(0x4124007d23c0) 0x1a, CmdSN 0x1f38 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-01-17T14:22:23.876Z cpu0:4104)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124007d23c0, 0) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T8:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-01-17T14:22:23.876Z cpu0:4104)ScsiDeviceIO: 2316: Cmd(0x4124007d23c0) 0x1a, CmdSN 0x1f3a from world 0 to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-01-17T14:24:54.127Z cpu2:7914)Config: 347: "VMOverheadGrowthLimit" = -1, Old Value: 0, (Status: 0x0)
2013-01-17T14:26:05.551Z cpu5:5120)Config: 347: "VMOverheadGrowthLimit" = 0, Old Value: -1, (Status: 0x0)
2013-01-17T14:26:06.331Z cpu7:7841)Config: 347: "HostLocalSwapDirEnabled" = 0, Old Value: 0, (Status: 0x0)
2013-01-17T14:27:23.867Z cpu4:7892)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400804800, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-01-17T14:27:23.867Z cpu4:7892)ScsiDeviceIO: 2316: Cmd(0x412400804800) 0x1a, CmdSN 0x1f77 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-01-17T14:27:23.890Z cpu1:4097)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400804800, 0) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T8:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-01-17T14:27:23.890Z cpu1:4097)ScsiDeviceIO: 2316: Cmd(0x412400804800) 0x1a, CmdSN 0x1f79 from world 0 to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-01-17T14:31:42.052Z cpu6:5167)Config: 347: "VMOverheadGrowthLimit" = -1, Old Value: 0, (Status: 0x0)
2013-01-17T14:32:23.880Z cpu0:4104)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124007ed380, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-01-17T14:32:23.880Z cpu0:4104)ScsiDeviceIO: 2316: Cmd(0x4124007ed380) 0x1a, CmdSN 0x1fd4 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-01-17T14:32:23.907Z cpu4:4100)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124007ed380, 0) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T8:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-01-17T14:32:23.907Z cpu4:4100)ScsiDeviceIO: 2316: Cmd(0x4124007ed380) 0x1a, CmdSN 0x1fd6 from world 0 to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
hostd.log
2013-01-17T14:33:42.072Z [6AEC8B90 warning 'PropertyCollector'] ComputeGUReq took 1958499 microSec
2013-01-17T14:33:43.182Z [6AA2AB90 verbose 'Default' opID=015AD5D3-0000029E-73] CloseSession called for session id=0ef5e3f6-7424-d0b0-fda8-04104e160e35
2013-01-17T14:33:43.183Z [6AA2AB90 info 'ha-eventmgr' opID=015AD5D3-0000029E-73] Event 293 : User vpxuser@140.247.49.144 logged out (login time: Friday, 02 January, 1970 01:15:55, number of API invocations: 0, user agent: VMware-client/5.1.0)
2013-01-17T14:33:53.696Z [6AA2AB90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T14:34:03.715Z [FF962D20 verbose 'Default'] Timed out reading between HTTP requests. : Read timeout after approximately 50000ms. Closing stream TCP(local=127.0.0.1:8309, peer=127.0.0.1:52321)
2013-01-17T14:34:03.715Z [FF962D20 verbose 'Default'] Timed out reading between HTTP requests. : Read timeout after approximately 50000ms. Closing stream TCP(local=127.0.0.1:8307, peer=127.0.0.1:59344)
2013-01-17T14:34:03.715Z [6AB2EB90 error 'HttpSvc.HTTPService'] Failed to read request; stream: TCP(<null>), error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2013-01-17T14:34:03.716Z [6AE86B90 error 'SoapAdapter.HTTPService'] Failed to read request; stream: TCP(<null>), error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2013-01-17T14:34:04.502Z [6AE05B90 verbose 'Locale' opID=QS-host-537-c9d462e1-49] Default resource used for 'counter.sys.resourceMemConsumed.summary' expected in module 'perf'.
2013-01-17T14:34:04.540Z [6AD84B90 verbose 'ha-license-manager' opID=QS-host-537-c9d462e1-49] Load: Loading existing file: /etc/vmware/license.cfg
2013-01-17T14:34:04.558Z [6AD84B90 verbose 'Default' opID=QS-host-537-c9d462e1-49] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)
2013-01-17T14:34:04.568Z [6AD84B90 error 'Default' opID=QS-host-537-c9d462e1-49] Unable to parse MaxRam value:
2013-01-17T14:34:04.568Z [6AD84B90 error 'Default' opID=QS-host-537-c9d462e1-49] Unable to parse MaxRamPerCpu value:
2013-01-17T14:34:04.569Z [6AD84B90 error 'Default' opID=QS-host-537-c9d462e1-49] Unable to parse MinRamPerCpu value:
2013-01-17T14:34:04.569Z [6AD84B90 error 'Default' opID=QS-host-537-c9d462e1-49] Unable to parse vram value:
2013-01-17T14:34:05.871Z [6AAEDB90 info 'TaskManager' opID=QS-host-537-c9d462e1-e1] Task Created : haTask-ha-host-vim.option.OptionManager.updateValues-514232047
2013-01-17T14:34:05.882Z [6AAEDB90 info 'TaskManager' opID=QS-host-537-c9d462e1-e1] Task Completed : haTask-ha-host-vim.option.OptionManager.updateValues-514232047 Status success
2013-01-17T14:34:05.915Z [FF962D20 info 'TaskManager' opID=QS-host-537-c9d462e1-e1] Task Created : haTask-ha-compute-res-vim.ComputeResource.reconfigureEx-514232049
2013-01-17T14:34:06.084Z [6AD84B90 verbose 'NetConfigProvider'] FetchFn: List of pnics opted out
2013-01-17T14:34:06.643Z [6AD84B90 verbose 'Default'] StorageSystemVmkImplProvider: advanced option get key = VMFS.UnresolvedVolumeLiveCheck
2013-01-17T14:34:06.673Z [6AA2AB90 info 'ComputeResource' opID=QS-host-537-c9d462e1-e1] Updated local swap datastore policy: false
2013-01-17T14:34:06.673Z [6AA2AB90 info 'TaskManager' opID=QS-host-537-c9d462e1-e1] Task Completed : haTask-ha-compute-res-vim.ComputeResource.reconfigureEx-514232049 Status success
2013-01-17T14:34:06.944Z [6A9C9B90 verbose 'NetConfigProvider'] FetchFn: List of pnics opted out
2013-01-17T14:34:07.504Z [6A9C9B90 verbose 'Default'] StorageSystemVmkImplProvider: advanced option get key = VMFS.UnresolvedVolumeLiveCheck
2013-01-17T14:34:08.717Z [FF962D20 verbose 'Default'] Timed out reading between HTTP requests. : Read timeout after approximately 50000ms. Closing stream TCP(local=127.0.0.1:8307, peer=127.0.0.1:54793)
2013-01-17T14:34:08.718Z [6AA2AB90 error 'SoapAdapter.HTTPService'] Failed to read request; stream: TCP(<null>), error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2013-01-17T14:34:18.371Z [6AE05B90 info 'SysCommandPosix' opID=015AD5D3-000002E0-46-14] ForkExec(/usr/bin/sh) 74694
2013-01-17T14:34:23.698Z [6AD84B90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T14:34:24.996Z [6AAEDB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2013-01-17T14:34:40.114Z [FF962D20 verbose 'Locale' opID=015AD5D3-0000029E-73-b5] Default resource used for 'counter.sys.resourceMemConsumed.summary' expected in module 'perf'.
2013-01-17T14:34:53.702Z [6AAEDB90 verbose 'SoapAdapter'] Responded to service state request
the login time looks odd 2 Jan 1970! but time is correct omn the server.
between that timeframe I have a gap!
2013-01-17T13:07:46.917Z cpu5:7914)Config: 347: "HostLocalSwapDirEnabled" = 0, Old Value: 0, (Status: 0x0)
2013-01-17T13:12:23.641Z cpu5:4101)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400769600, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
hostd.log
2013-01-17T13:08:04.709Z [FF962D20 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:08:15.918Z [6AE86B90 info 'SysCommandPosix' opID=D763F63B-000014C7] ForkExec(/usr/bin/sh) 66736
2013-01-17T13:08:28.009Z [6AE05B90 verbose 'Default'] Timed out reading between HTTP requests. : Read timeout after approximately 50000ms. Closing stream TCP(local=127.0.0.1:8307, peer=127.0.0.1:49557)
2013-01-17T13:08:28.009Z [6AE05B90 verbose 'Default'] Timed out reading between HTTP requests. : Read timeout after approximately 50000ms. Closing stream TCP(local=127.0.0.1:8309, peer=127.0.0.1:56426)
2013-01-17T13:08:28.009Z [6AD84B90 error 'SoapAdapter.HTTPService'] Failed to read request; stream: TCP(<null>), error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2013-01-17T13:08:28.009Z [6AEC8B90 error 'HttpSvc.HTTPService'] Failed to read request; stream: TCP(<null>), error: N7Vmacore16TimeoutExceptionE(Operation timed out)
2013-01-17T13:08:28.830Z [6AA6BB90 info 'Vmomi' opID=D763F63B-00001487] Activation [N5Vmomi10ActivationE:0xffc5d538] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:session[671eee21-561b-acbf-d723-6ebf99980032]52f70510-8
2013-01-17T13:08:28.830Z [6AA6BB90 verbose 'Vmomi' opID=D763F63B-00001487] Arg version:
--> "142"
2013-01-17T13:08:28.831Z [6AA6BB90 info 'Vmomi' opID=D763F63B-00001487] Throw vmodl.fault.RequestCanceled
2013-01-17T13:08:28.831Z [6AA6BB90 info 'Vmomi' opID=D763F63B-00001487] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2013-01-17T13:08:28.831Z [6A9C9B90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(error:Transport endpoint is not connected) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2013-01-17T13:08:34.710Z [6AE86B90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:08:42.885Z [6AAEDB90 error 'EnvironmentManager'] Unable to obtain the default HW version
2013-01-17T13:08:42.899Z [6AAACB90 verbose 'DvsTracker'] FetchSwitches: added 0 items
2013-01-17T13:08:42.899Z [6AAACB90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
2013-01-17T13:08:42.899Z [6AAACB90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 0 items
2013-01-17T13:09:03.845Z [6AB2EB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2013-01-17T13:09:04.711Z [6AE05B90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:09:05.387Z [6AD84B90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 14423MHz/13571MB to 14423MHz/13572MB
2013-01-17T13:09:34.713Z [6AAEDB90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:10:02.437Z [6AEC8B90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:10:02.540Z [6AAACB90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0xffccc458] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
2013-01-17T13:10:02.540Z [6AAACB90 verbose 'Vmomi'] Arg version:
--> "38"
2013-01-17T13:10:02.540Z [6AAACB90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2013-01-17T13:10:02.540Z [6AAACB90 info 'Vmomi'] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2013-01-17T13:10:02.541Z [6AB2EB90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(error:Transport endpoint is not connected) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2013-01-17T13:10:04.714Z [6AAACB90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:10:05.388Z [6AEC8B90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 14423MHz/13572MB to 14423MHz/13571MB
2013-01-17T13:10:11.702Z [6A9C9B90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
2013-01-17T13:10:34.715Z [6AB2EB90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:10:35.290Z [FF962D20 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2013-01-17T13:11:04.717Z [FF962D20 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:11:05.390Z [6AE86B90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 14423MHz/13571MB to 14423MHz/13572MB
2013-01-17T13:11:34.718Z [6AB2EB90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:12:04.719Z [6AAEDB90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:12:06.734Z [6AD84B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2013-01-17T13:12:09.398Z [6AD84B90 verbose 'DvsManager'] PersistAllDvsInfo called
2013-01-17T13:12:34.721Z [6AE05B90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:13:04.722Z [6AA2AB90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:13:11.706Z [6AAACB90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
2013-01-17T13:13:34.723Z [6AD84B90 verbose 'SoapAdapter'] Responded to service state request
2013-01-17T13:13:38.178Z [6AA6BB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2013-01-17T13:13:42.936Z [6A9C9B90 error 'EnvironmentManager'] Unable to obtain the default HW version