Archive | Backup/Restore RSS for this section

Azure Recovery Services “Invalid Vault Credentials”

I was recently setting up Azure Recovery Services for a customer and was struggling to install the Azure Vault Credentials during the “Register Server” step.  Each time I would install I received the following error message:

2016-05-13_9-47-00

Invalid vault credentials provided. The file is either corrupted or does not have the latest credentials associated with recovery service. (ID: 34513)
We recommend you download a new vault credentials file from the portal and use it within 2 days.

I reviewed the Azure Recovery Services logs (C:\Program Files\Microsoft Azure Recovery Services Agent\Temp) and found some mostly generic .NET errors about being unable to communicate with the server.  I attempted to connect via the Azure Backup PowerShell cmdlets which returned the following error:

Login-AzureRmAccount : The browser based authentication dialog failed to complete. Reason: The request could not be processed by the server due to invalid syntax.

On a bit of a hunch, I looked into the Security Providers settings for the server to see if any client protocols and been adjusted and noticed that they had all been hardened and a lot of ciphers client ciphers were disabled beyond the defaults.
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols

2016-05-13_14-41-43

I compared these settings against another server and then removed all of the additional entries back to the defaults (after backing up the changes):

2016-05-13_14-55-41

After that my Azure PowerShell connection worked perfectly:

2016-05-13_14-57-44

From there I had to remove all of the old machine certificates added through all of the failed attempts…

2016-05-13_15-04-06

Finally a re-install of the agent and a re-register of the server completed successfully.

2016-05-13_15-14-36

Veeam: Failed to prepare (Hyper-V) guests for volume snapshot

After a few big months (we recently moved back to the U.S. for a couple of years) – I thought I better get back to some blogging!  Here is a recent issue I’ve ran into for a customer while setting up a new Veeam environment to backup a Hyper-V cluster:

Host and guest O/S both Server 2012 R2.  When backing up a specific VM I received the following errors:

Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent). Details: Failed to prepare guests for volume snapshot.
Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent). Details: Failed to prepare guests for volume snapshot.
Processing servername Error: Failed to prepare guests for volume snapshot.

2016-01-04_10-28-20

I checked the VSS Writers, etc. and confirmed they were all good.  Checked to ensure there was snapshot space available.  There was no need to upgrade the Integration services as suggested.  After doing a little further digging, I noticed that the Hyper-V integration services components were set to disabled in the Guest O/S.  This was in spite of the properties for the VM stating otherwise:

2016-01-04_10-26-38
2016-01-04_10-27-59

The fix was to set all of the services to manual (triggered start) & reboot the VM.

2016-01-04_10-27-34

Once this was completed backup ran successfully!

 

Bulk Fortigate configuration backup

Continuing on from the previous post with bulk management of firewalls, in particular Fortigates.  The following shows a basic script which after enabling SCP transfers on your device, utilizes pscp.exe to backup multiple Fortigate firewall configs.

This script gets the current date, creates a folder under your Backup Path with the date, connects to each firewall in your CSV file (just as the previous post) and uses pscp.exe to backup the config file (fgt-config).  It then renames the fgt-config output file to the IP of your firewall and appends the date to the file name.  I am utilizing the echo y command at the front of the command to accept the untrusted certificate error at first connection.  In a production environment you may wish to re-evaluate this option!

Note, while the transcript option is present, due to a bug in PowerShell you do not get a whole lot of info out of running cmd applications from PowerShell.  I’ll look at a work around for this and update in the near future.

backup-config2
————————————————————————————
$Firewalls = get-content .\firewalls.csv
$Date = Get-Date -Format “dd-MM-yyyy”
$Username = “firewallusername”
$Password = “firewallpassword”
$BackupPath = “E:\Firewall-Config-Backups”
New-Item $BackupPath\$Date\ -type directory
Start-Transcript -path $BackupPath\$Date\$Date.log -append

foreach ($Firewall in $Firewalls){

echo y | .\pscp.exe -l $Username -pw $Password -v “$Firewall::fgt-config” $BackupPath\$Date\ | out-host -verbose
rename-item -path $BackupPath\$Date\fgt-config -newname $BackupPath\$Date\$Firewall-$Date.conf

}

Stop-Transcript
————————————————————————————

Here’s what your final screen output should look like:

backup-config-new

Veeam: A VSS critical writer has failed

While deploying Veeam Backup & Replication recently for a customer, we came across the following error message while backing up a domain controller:

Unable to release guest. Error: Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer’s data.
A VSS critical writer has failed. Writer name: [NTDS]. Class ID: [{b2014c9e-8711-4c5c-a5a9-

3cf384484757}]. Instance ID: [{bcc3ddec-3a1b-4bbd-bb5c-2da62e9bcb4d}]. Writer’s state: [VSS_WS_FAILED_AT_POST_SNAPSHOT]. Error code: [0x800423f4].]
veeamerror

Running VSSAdmin List Writers showed in fact that the NTDS Writer was in an Error state with [11] Failed, “Non-retryable error”.

The issue was due to a conflict with Shadow Protect, which was also doing VSS Backups.  As Veeam was going to be replacing ShadowProtect, the fix for this was to remove the Shadow Protect VSS Provider:

First re-run the ShadowProtect backup which restored the NTDS Writer to a Ready State.
veeamerror1

Open Regedit
Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\
You should see the GUID {24602736-bed9-4619-91b0-243447c6409c} listed.  Clicking on it shows “StorageCraft Volume Snapshot Software Provider”
veeamerror3
Backup this registry key, then delete it.
veeamerror2
Open Windows services and restart the “Volume Shadow Copy” service. It may be stopped to begin with, that’s okay.
Check and see if the provider is gone from: vssadmin list providers
Reboot the server when next possible.

Re-run Veeam backup and all should be successful!

RichCopy – Advanced File Copy Utility

We recently had a customer with a nasty piece of malware that corrupted all of their .xlsx & .docx files.  Apparently a new piece of “ransomware” Win32/Crilock.A and Win32/Buma!rts that are floating around at the moment.  They utilized ShadowProtect for their image backups which was great as we were able to recover their data to just prior to the infection.  The difficulty with this is that their docx & xlsx files were spread through 100’s of folders and simply restoring the volume wouldn’t work as we needed to filter by file extension.  Thankfully there is a great tool (formally an internal tool) from Microsoft which is an extremely feature-rich copy utility & best of all, free!  RichCopy at first glance appears to be a decent file copy program like RoboCopy, TeraCopy, etc.

RichCopy1

The real magic begins once you enable the “Advanced” functionality under the “View” menu:

RichCopy2

From here you see a ton more features available under the Options screen:

RichCopy4

These include allow you to filter by file type, set throttling for transfer, change file attributes, adjust security information, issues commands, etc.