Hi folks,
We are a Cloud Provider who uses CommVault as the primary backup solution for our virtualized and physical workloads. Out whole backup environment is virtual as well including Media Agents and VSA Proxies. These VSA Proxies are just a W2K8 R2 server that is used to HotAdd snapshots for the backup process.
We have always had issues with the VSA Proxies around the HotAdd process with typical issues in having random HotAdd failures. Our original issue revolved around the fact that when HotAdd is required to mount disks past the 1st controller it would fail with the HotAdd. Our VSA Proxies right now have 3 controllers so I would assume we should be able to mount 45 disks. We had solved the issue in the past without really knowing how but after a recent upgrade/migration, it appears we have issues again.
CommVault support has not really been very helpful with this issue and after much searching I am basically at the end of my rope.
I suspect that the issue may be centered around the order or binding of the adapters being different than how they are represented in VMware. I have come to this forum as this all revolves around VDDK and the interworkings of VMware.
Technically I should be able to use 4 controllers but to be honest, if we could get 3 working without issues I would be happy.
We are running vSphere 5.0 U1 with all datastores now being Version 5.
What I would really love is an explaination of the events that are logged in the VixDisk log in CommVault with the hopes that it sheds some light on this issue. SImple things like:
1.. 2444 474 12/05 01:09:30 ### diskLibPlugin: 2012-12-05T01:09:30.516-05:00 [01140 verbose 'Default'] ScsiEnum_GetAdapterPort: scsi port \\.\Scsi0: bus type is 3
2444 474 12/05 01:09:30 ### diskLibPlugin: 2012-12-05T01:09:30.516-05:00 [01140 verbose 'Default'] ScsiEnum_GetAdapterPort: scsi port \\.\Scsi1: bus type is 3
2444 474 12/05 01:09:30 ### diskLibPlugin: 2012-12-05T01:09:30.516-05:00 [01140 verbose 'Default'] ScsiEnum_GetAdapterPort: scsi port \\.\Scsi2: bus type is 10
Why is the 3rd Adapter showing with bus type 10 when they are the same?
2.. CreateFile \\.\PhysicalDrive15 failed 2
Are these normal? We will have 15 through 64 typically here for this failure
3.. TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.
We have groups of 3 of these in the logs as well. Could this be part of the problem?
4.. Mounting VM vim.VirtualMachine:vm-85249 using transport hotadd failed : Failed to hot-add SCSI targets: Vmomi::MethodFault::Exception: vim.fault.GenericVmConfigFault.
This is a HotAdd failure. It was trying to mount the drives at 1:3 1:4 1:5 but failed. If I run this job manually after te fact by itself, no issues. I seems to be due the fact that I am HotAdding many VM drives to the VSA Proxy.
At this point in time, any advice would be helpful.
Cheers,