A. Mikkelsen

VMware ESX scripts, commands, tools and other nice to know things that will make your virtualization days easier!!!!

Browsing Posts in Disaster Recovery

If you have your hosts connected to a Cisco network infrastructure, you can see a hosts CDP information directly from within the VI Client.

As you can see in the exampels below it’s quite impossible to get the full CDP picture if you have many hosts with multiple NICs.
So I created a script that retrieves all CDP info from all your hosts (even across multiple vCenters) and displays it as a webpage.
Now it’s possible to search and share the information 🙂

The script is build up by a few functions

  • Retrieve the vCenter servers to retrieve host from.
    $objvCenterServer = Import-Csv -Path $strvCenterFilePath -Delimiter ";" | sort vCenter
    foreach($strvCenterServer in $objvCenterServer){
    # Check if VC is uncommented
    if ((!($strvCenterServer.vCenter.Contains("#"))) -and ($strvCenterServer.vCenter.Length -gt 0)){
    # Connect to vCenter Server
     Connect-VIServer -Server $strvCenterServer.vCenter -User $strvCenterServer.UserName -Password $strvCenterPWD
    
    # Add logic
    
    # Disconnect from vCenter server
    DisConnect-VIServer -Confirm:$false
    }}
    
  • Retrieve hosts from clusters.
    $arrDC = Get-Datacenter | Sort
    foreach($objDC in $arrDC){
     $arrCluster = Get-Cluster -Location $objDC | Sort
     # Only proceed if the Cluster isn't blank
     if ($($arrCluster | Measure-Object).count -gt 0){
     foreach($Cluster in $arrCluster){
     $vmhosts = Get-VMHost -Location $Cluster | Sort Name | Where-Object {$_.State -eq "Connected"} | Get-View
     #Only proceed if any hosts in cluster
     if ($vmhosts.Count -gt 0){
     foreach ($vmhost in $vmhosts){
    
    # Add logic for each host
    
    }}}}}
    
  • Retrieve hosts NICs CDP info.
    foreach ($vmhost in $vmhosts){
     $networkSystem = Get-view $vmhost.ConfigManager.NetworkSystem
    
     foreach($pnic in $networkSystem.NetworkConfig.Pnic | Sort Device){
     $pnicInfo = $networkSystem.QueryNetworkHint($pnic.Device)
    
     foreach($Hint in $pnicInfo){
    
     # LinkSpeed & MAC
     $record = 0
     $tmpSpeed = ""
     $tmpMAC = ""
     Do{
     If ($Hint.Device -eq $vmhost.Config.Network.Pnic[$record].Device){
     $tmpSpeed = $vmhost.Config.Network.Pnic[$record].LinkSpeed.SpeedMb
     $tmpMAC = $vmhost.Config.Network.Pnic[$record].Mac
     }
     $record ++
     }
     Until ($record -eq ($vmhost.Config.Network.Pnic.Length))
    
     # Duplex
     $tmpDuplex = ""
     if($Hint.ConnectedSwitchPort.FullDuplex -eq $true){
     $tmpDuplex = "Full"
     }
     if($Hint.ConnectedSwitchPort.FullDuplex -eq $false){
     $tmpDuplex = "Half"
     }
    
     # Status
     $tmpStatus = ""
     If (($tmpSpeed -ge 1000) -and ($tmpDuplex -eq "Full")){
     $tmpStatus = "OK"
     }else{
     If (($tmpSpeed -gt 0) -and ($tmpDuplex -eq "")){
     $tmpStatus = "CDP not working"
     }
     elseif (($tmpSpeed -gt 0) -and ($tmpDuplex -eq "Half")){
     if ($tmpStatus -ne ""){$tmpStatus += " / "}
     $tmpStatus = "Duplex config error"
     }
     elseif (($tmpSpeed -gt 0) -and ($tmpSpeed -lt 1000)){
     if ($tmpStatus -ne ""){$tmpStatus += " / "}
     $tmpStatus = "Speed config error"
     }
     else{
     $tmpStatus = "Link Down"
     }}}}
    
  • Seperate information included/excluded.
    $isExcluded = func_exclude_from_list -ExcludeHost $vmhost.Name -ExcludeVMNIC $Hint.Device -FromObj $objExcludeList
    
  • Generate html outputfile.
    Add-Content -Path $($strOutputPath + $strOutputFileName + ".TMP") -Value $strHTML
    if(Test-Path -Path $($strOutputPath + $strOutputFileName + ".htm")){
     Copy-Item -Path $($strOutputPath + $strOutputFileName + ".htm") -Destination $($strOutputPath + $strOutputFileName + $(Get-Date -uformat "%Y%m%d") + ".htm") -Force
    }
    Copy-Item -Path $($strOutputPath + $strOutputFileName + ".TMP") -Destination $($strOutputPath + $strOutputFileName + ".htm") -Force
    Remove-Item -Path $($strOutputPath + $strOutputFileName + ".TMP") -Force
    

vCenter servers are added/removed from the file “_All_vCenter_Hosts_.csv” .
A hosts NICs are moved to the exclude section, by adding it to the host exclude list “host_CDP_exclude.csv”.

The CDP information is sorted and saved in a HTML output file. If the script has been run before and an older version of the output file exist, it’s renamed and the new is saved (This supply you with a sort of history).

Get all script files here.
_All_vCenter_Hosts_.csv
host_CDP_exclude.csv
host_cdp_info_v01.ps1

————————-

For CDP information to be visible  in the VI client, CDP must also be enabled/configured on your physical Cisco switch – see example.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003885

To enable/change the CDP information on a host you have several options.

  1. If you are using dvSwitches in vSphere 4.x you can enable it from GUI .

    • Connect to vCenter using VI Client
    • From Home click Networking
    • Right click your dvSwitch and select Edit Settings
    • Under the Properties tab select Advanced
    • Check Cisco Discovery Protocol
    • Set Operation to Both
    • Click OK
  2. Using ESX Command line
    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003885
    (exchange vSwitch1 with the name of your vSwitch)

    • Show current setting
      esxcfg-vswitch -b vSwitch1
    • Set the new status (down, listen, advertise, both)
      esxcfg-vswitch -B both vSwitch1
    • Verify new settings
      esxcfg-vswitch -b vSwitch1
  3. Using vMA
    http://spininfo.homelinux.com/news/vSphere_PowerCLI/2010/03/24/Enable_CDP_on_vSwitch_on_all_hosts

     vicfg-vswitch --server <vcenter.domain.com> -h <esxi.domain.com> -B both <vSwitch1>

To view your CDP info you also a few options
See all options here:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007069

  1. GUI

    • Connect to vCenter or ESX using VI Client
    • Select a ESX host
    • Click the Configuration tab
    • Select Networking
    • Click the Info icon to the right of the vSwitch
    • A tool tip opens with CDP information for the selected physical network interface
  2. PowerShell
     Get-VMHost | Where-Object {$_.State -eq "Connected"} |
     %{Get-View $_.ID} |
     %{$esxname = $_.Name; Get-View $_.ConfigManager.NetworkSystem} |
     %{ foreach($physnic in $_.NetworkInfo.Pnic){
        $pnicInfo = $_.QueryNetworkHint($physnic.Device)
        foreach($hint in $pnicInfo){
           Write-Host $esxname $physnic.Device
           if( $hint.ConnectedSwitchPort ) {
              $hint.ConnectedSwitchPort
           }
           else {
              Write-Host "No CDP information available."; Write-Host
           }
        }
     }
    

Just released the complete powershell script to control vRanger backup from vCenter.

Read about the script and it’s features here.

Download it here.

Since Vizioncore came out with vRanger 4.x, last year, I been working on porting the VBS script “vRanger Script version 2.0” to Powershell.
During this time I have found several bugs/missing functionality in the vAPI.
Vizioncore have corrected the bugs with new releases of vRanger.
The first vRanger version where all vAPI cmdlets worked in was version 4.2.3.

Porting the script from VBS presented me with a few challenges.

  • A reported error in the Get-Date cmdlet in Powershel 2.0.
    Had to use a workaround.

A few words on the scrips functionality.

  • Control vRanger backups from vCenter (simple – one interface).
  • Perform a FULL backup of DiskO of a VM.
  • Perform a FULL backup of all disks of a VM.
  • Backup a VM on a specific weekday.
  • Control what week (Even/Odd).
  • Create a log file.
  • Delete previous backup files from repository.
  • Delete previous savepoints.
  • Send an email when the jobs are completed – incl. succes/errors.
  • Offload the files to tape – TSM
  • Send mail when TSM is finished offloading backup files.
  • And many more…..

When I started coding the script, I found, that if I used functions for all functionality the script was easier to read. Using functions also supplied me with the opportunity of reusing functions – less work :-).

Some of the functions was created using examples posted and mailed by Scott Harold from Quest and TheVesi./vEcoShell.

Prerequirements

The script can be downloaded here.

A quick overview of the functions.
Include the right PSSnapins

function LoadSnapin{
    Add-PSSnapin vRanger.API.PowerShell -ErrorAction SilentlyContinue
    Add-PSSnapin vmware.VimAutomation.core -ErrorAction SilentlyContinue
}

LoadSnapin

Delete all previous savepoints – from the repository.

function RemoveAllSavePoints{
	param ($RASP_RepoName)
	$RASP_repoID = Get-Repository | where {$_.Name -eq $RASP_RepoName}
	$RASP_SPlist = Get-RepositorySavePoint -ID $RASP_repoID.Id

	foreach ($RASP_SP in $RASP_SPlist){
		#write $RASP_SP
		Remove-SavePoint -SavePointsToRemove $RASP_SP
	}
}

RemoveAllSavePoints "Remote-Location"

Is a backup job exist with the same name as the one to create – delete it.

function Del-vRangerBackupJobs {
	param ($delJobName)
	$delTemplateID = Get-JobTemplate | where {$_.JobName -eq $delJobName}
	# Verify that a BackupJob named XXX is present before deleting it.
	if ($delTemplateID){
		#write $delTemplateID.JobName
		Remove-JobTemplate -id $delTemplateID.Id
	}
}

Del-vRangerBackupJobs "$Wednesday-Even"

Create the list of VMs to exclude.

function Filter-VMbyCF {
    param ($customF, $val)
    foreach ($vm in $vmlist) {
        $vm.CustomFields | ForEach-Object {
            $cf = $_
            if ($_.Key -like $customF -and $_.Value -like $val){
                    return $vm.name
            }
        }
    }
}

function New-vRangerExcludeList {
    param ($exclude, $include)
    $excludeArray = @()
    $exclude | ForEach-Object {
        if ($include -notcontains $_.Name){
            $excludeArray += ($_.Config.Uuid)
        }
    }
    return $excludeArray
}

$vmlist = Get-VM
$tmplist = Get-Template
$vmlist= $vmlist + $tmplist
$vmFilter = Filter-VMbyCF $customField $CFValue
$vmlistview = $vmlist | Get-View
$excludeList = New-vRangerExcludeList -exclude $vmlistview -include $vmFilter

Create the new backup job.

function New-vRangerBackupJob {
	param ($customJobName, $customCFValue, $customFieldName, $customRepoName, $customExcludeList, $customEmail)
        $jobDesc = "This backup job is created by Logica for use in DR.  All VMs with a Custom Field labeled: $customFieldName with a value of: $customCFValue will be backed up"
        $jobEntity = Get-InventoryEntity -Type VirtualCenter
	$jobRepos = Get-Repository | where {$_.Name -eq $customRepoName}
        $jobFlag = New-BackupFlag -CheckDestinationFreeSpace:$true -UseCompression:$true -PerformNetworkBackupOnFailure:$true

	# Check if all disks or just disk 0
	if ($customJobName -like "*-All*"){
		#write "All - Disks"
		#Add-BackupJobTemplate -JobName $customJobName -JobDescription $jobDesc -JobEntity $jobEntity -ExcludedVMList $customExcludeList -NotificationList $email -TargetRepository $jobRepos -Flags $jobFlag -NumberOfSavePoints 1 -SpaceSavingTechnologyTypeFlag None
		Add-BackupJobTemplate -JobName $customJobName -JobDescription $jobDesc -JobEntity $jobEntity -ExcludedVMList $customExcludeList -NotificationList $customEmail -TargetRepository $jobRepos -Flags $jobFlag -NumberOfSavePoints 1 -SpaceSavingTechnologyTypeFlag None -SpaceSavingCountThreshold 6 -SpaceSavingPercentSizeThreshold 50
	}
	else{
		#write "Only Disk 0"
		#Add-BackupJobTemplate -JobName $customJobName -JobDescription $jobDesc -JobEntity $jobEntity -ExcludedVMList $customExcludeList -NotificationList $email -IncludedDiskList 0 -TargetRepository $jobRepos -Flags $jobFlag -NumberOfSavePoints 1 -SpaceSavingTechnologyTypeFlag None
		Add-BackupJobTemplate -JobName $customJobName -JobDescription $jobDesc -JobEntity $jobEntity -ExcludedVMList $customExcludeList -NotificationList $customEmail -IncludedDiskList 0 -TargetRepository $jobRepos -Flags $jobFlag -NumberOfSavePoints 1 -SpaceSavingTechnologyTypeFlag None -SpaceSavingCountThreshold 6 -SpaceSavingPercentSizeThreshold 50
	}
}

New-vRangerBackupJob "Wednesday-Even" "Wednesday-Even" "Backup" "Remote-Location" $excludeList "to@email.com"

Start the backup job.

function Run-BackupJob{
	param ($runJobName)
	$runTemplateID = Get-TemplateID $runJobName
	Run-JobsNow $runTemplateID
}

Run-BackupJob "Wednesday-Even"

Wait for the backup job to finish.

function Get-TemplateID{
	param($tmpJobName)
	$arrTemplate = get-jobtemplate | where {$_.JobName -eq $tmpJobName}
	return $arrTemplate.ID
}

function WaitForJobToFinish{
	param ($strWaitJob)
	$strCompleted = "NotStarted"
        Start-Sleep -Seconds 300	# 5 minutes
	$jobTemplates = get-JobTemplate | where {$_.JobName -eq $strWaitJob}
	foreach ($job in $jobTemplates){
		do{
			$tasksinfo = get-job |where {$_.ParentJobTemplateID -eq $job.TemplateVersionID}
			foreach ($task in $tasksinfo){
				Start-Sleep -Seconds 300	# 5 minutes
				$strCompleted = $task.JobState

                # Make sure job is running - if not start it...
                if ($task.JobState -eq $NULL -or $task.JobState -eq "NotStarted"){
                    Run-BackupJob $strWaitJob
                    Start-Sleep -Seconds 120	# 2 minutes
                }
			}
		}
		while ($strCompleted -notmatch "Completed")
	}
}

WaitForJobToFinish "Wednesday-Even"

Send TSM finished mail.

function FuncMail {
	#param($strTo, $strFrom, $strSubject, $strBody, $smtpServer)
	param($To, $From, $Subject, $Body, $smtpServer)

	$msg = new-object Net.Mail.MailMessage
	$smtp = new-object Net.Mail.SmtpClient($smtpServer)
	$msg.From = $From
	$msg.To.Add($To)
	$msg.Subject = $Subject
	$msg.IsBodyHtml = 1
	$msg.Body = $Body
	$smtp.Send($msg)
}

FuncMail -To "to@email.com" -From "from@email.com" -Subject "vRanger Pro: TSM backup finished" -Body "Your Body" -smtpServer "your.mailserver.com"

A friend on the Danish VMUG pointed me towards this simple backup script, from GestaltIT.com.
http://gestaltit.com/all/tech/virtualization/simon/vmware-backup-powercli-script/

The script creates a snapshot of a VM (from a CSV file) and creates a Thin Provisioned clone based on the snapshot, on a specified datastore (from the CSV file).

This a smart and simple way to create a backup of a list of VM’s.

It’s a must read for PowerCLI users.

A fellow member in the Danish Usergroup has created a very simple backup script for ESX and ESXi using PowerShell and VCB.

http://www.virtualize.dk

I like that it’s small and simple.
Keep up the good work….

Yesterday I by mistake added the VirtualCenter servers local Users group to a folder in VC with ReadOnly permissions.

This resulted in that I couldn’t delete or change any permissions on the folder.

To solve the problem do the following (On a MSSQL2000 – should be almost the same on MSSQL2005):

  • Open the SQL Server Enterprice Manager and browse to the VC_DB (or what you have called the VC database)
  • Locate the table VPX_ACCESS and right click it and choose Open Table -> Return all rows
  • In the buttom of the table you should be able to locate the wrong permissions entery – make a note of the ID.
    If you can’t find the user/group, you can browse the VPX_ENTITY table to locate ENTITY_ID you need.
  • Open the SQL Query Analyzer an choose the VC database
  • To delete the row that contains the wrong permissions run the below SQL code or modify it for you liking.
    To only delete row 221 from the table

    DELETE FROM esx.VPX_ACCESS WHERE ID = 211

    To delete all rows containing a specific user/group from the table

    DELETE FROM esx.VPX_ACCESS WHERE PRINCIPAL = 'your_user or group'

After you have deleted or updated all the permissions you need, you have to restart the VC server service before the changes will take effect.

Disclamer…..
All changes to the VC database should be avoided at all time.
Always make a backup of the database before making ANY changes to it.
All changes to the VC database using examples on this website is at your own risk.

A few weeks ago we were conducting a vRanger DR test of a VM (new host and new LUN).

The VM was restored succesfull, but when we powered it on we discovered that it for some unexplaned reason had lost the SCSI0:1 (100GB) and SCSI0:2(300GB) VMDK’s.
It had created two new 20GB VMDK’s instead.
We searched the LUN and found the two orginal *-flatvmdk files but not the descripter files.

Use Putty to identify the size of the *-flatvmdk file. ex. 100GB

ls -lah

From a VM (not running) create a new disk with the same size as the one you are missing. The name is not important.

Locate the newly created *.vmdk and *-flat.vmdk file. Copy the new *.vmdk file to the folder that contains the orginal *-flat.vmdk file.

cp rescue_me.vmdk /vmfs/volumes/mysan/rescued/rescued.vmdk

Use VI or NANO to change the following line as below from:
RW 419426200 VMFS “rescue_me-flat.vmdk”

To:
RW 419430500 VMFS “rescued-flat.vmdk”

Make sure that the name of the .vmdk file correspond to the SCSI*:*.fileName in the vmx file.
Now just power on the VM and the orginal disks are intac.

UPDATE
Today I found out that esXpress has created a website that can create a VMDK descriptor file.
http://www.esxpress.com/tools/wrapgen.php

Below is an exampel for the rescure_me-flat.vmdk file with a size of 100GB (107374182400 bytes)
esXpress WMDK descriptor file creator

“This first VMware® VMbook focuses on Business Continuity and Disaster Recovery (BCDR) and is intended to guide the reader through the step-by-step process to set-up a multisite VMware Infrastructure that is capable of supporting BCDR services.

The VMbook will provide very valuable insight into the considerations and design principles for a multisite virtual infrastructure data center that leverages array based replication for the replication of VMFS datastores; which is one of the prerequisites for Site Recovery Manager 1.0. The VMbook authors suggest your customers leverage this VMbook as a reference even if they are planning to implement a Site Recovery Manager based BCDR solution day one, as the BCDR solution detailed in this VMbook provides design principle guidance for a successful multisite virtual infrastructure data center deployment which is the underpinnings for a successful SRM deployment.”

The VMbook is available for download from: http://www.vmware.com/resources/techresources/1063

I’m now done with the new script and it’s working perfectly.

Background for the script:

At work we use vRanger to take DR snapshots of all our VM’s (more than 400).

We decided that we would only snapshot drive 0 and use a TSM client to backup the data in each VM and that the snapshots were only to be taken outside working hours.
These choices gave us some problems when running vRanger because we couldn’t schedule the snapshots from vRanger.

Because we have more than 400 VM’s we had to divide the list of witch VM’s to backup, into 14 days.
This means that one specific VM will only be snapshotted every 2 weeks.

In Virtual Center we created a custom field (=Backup) to tell the script witch VM’s to snapshot witch days.
We use full weekdays together with Even / Odd for dividing the VM’s into 14 days.
Ex. “Monday-Even” or “Sunday-Odd”

The solution was to make our own script that would handle the logic and just use vRanger to do the actual snapshotting.
We created a VBS script to hold the logic.
1.
Create a log file
2.
Delete snapshots from the day before (due to lack of storage space on the server)
3.
Generate the .cmd file from information in a custom field (Backup) in VC
4.
Call the .cmd file that was created earlier and contains the information on which VM’s to snapshot (Based on weekday – one file for each day).
5.
Start TSM (send the VM’s snapshots to tape)
6.
Send a status mail

(You have to use vRanger GUI to choose which drives to snapshot – changed from 3.17 -> 3.20)

I know this solution isn’t very dynamic but i works.

You can download the new script here – version 2.0.

Duncan over at YellowBricks.com found a undocumented fearure in the VCB config.js file

One of my customers wanted to use the default VCB framework but did not want to quiesce the VM for several reasons. (Databases, Active Directory etc.) I could not find an option in the config.js file but noticed the following in the file glue.js:

// A fallback to be able to switch to non-quiesced snapshots
if (typeof(NO_QUIESCE) != "undefined") {
cmd +="-Q 0 ";
}

In other words, setting the option “NO_QUIESCE” with no value in config.js results in the VM not being quiesced, default it will quiesce the VM! I added the following line to the  config.js file to accomplish this:

NO_QUIESCE="";

Powered by WordPress Web Design by SRS Solutions © 2018 A. Mikkelsen Design by SRS Solutions