Emergency SAN Data Recovery Advice

[Enough]

Let's say that there was a really bad icky no damn good IT incident that results from when folks go bad. The impact was severe risk of data loss and said data is not just commercial widget data, but really valuable research that besides representing $50 million+ in research, but sadly is also irreplaceable loss for society in the greater good sense. Now let's continue this very dark fantasy. What if you had a RAID 5 SAN that a rogue deleted all the vdisks on as part of that nuclear strike and you were tasked with working miracles. Would you go with DriveSavers or Ontrack? Yes we have quotes from both ($20k+ but it's a no-brainer), but what say you OO? May the force be with you.

[Jeff V]

When something similar happened years ago on a company server in Fairfax, VA, the case was investigated by the Secret Service (they were the cyber crime authority in the DC area). Had I not already taken expensive steps at restoring the data as best we could, they said they could have arranged recovery using non-commercially available tools. If you have any competent law enforcement investigating, they might have suggestions.

[LordMortis]

Call the Fed was my first thought as well. That's a straight to legal question. I'm going to have nightmares tonight.

[naednek]

this data wasn't backed up elsewhere?

[Enough]

this data wasn't backed up elsewhere?

Of course it was, but being a nonprofit my insistence that we have a third backup for proper 321 was never adequately funded and so backups were relying on a single Veritas Backup Exec backup and it was part of the incident. Thankfully, after nearly no sleep for a few days last Fri-Sun I was able to resurrect some, but the best I can do is back to April for the network drive shares and thankfully I can get early Aug for the web, db and GIS servers (this also took out 90% of our servers, luckily the DC, backups and file servers survived mostly). Make no mistake this was an attempt to nuke us from orbit, but we are not going to let that happen. I do have the entire campus IT community pitching in and actually have not touched the SAN since it got nuked to increase chances of professional data recovery. And the rest of the management team is certainly now onboard to fund the third backup and other IT enhancements we've been needing. I am taking full advantage of this to make sure we will end up stronger and more resilient in IT than we ever have been before.

Very few labs around the country can do the SAN recovery but the two I listed came recommended from a colleague at VMWare who has used both a number of times for similar issues successfully. The key is no writes and I think (fingers crossed) we are good there. We won't know until we can send in the drives to be evaluated in the lab but I can't do that until the insurance and police reports are complete. Then it's likely a couple of weeks to a month before we get that back. I just wanted to check hoping the OO Hive Mind may have faced a similar issue before and had opinions on DriveSavers and Ontrack/Kroll or other options. I have also of course scoured the web for said opinions. Thanks all!

[gbasden]

I had a very good experience in a similar situation with Ontrack, but it was about 7 or 8 years ago. I'm not sure how applicable that is at this point.

[FishPants]

Ontrack is one of the best -- I would send it there. Also fuck that guy that did that - hope he/she was arrested?

[Enough]

Thanks all, the team decided to go with DriveSavers and they got the drives on Friday for evaluation. I spoke to a fairly heavy user of both DriveSavers and OnTrack from VMWare who said he has had equally good luck with both and it looked like they would have a quicker turnaround. Fingers crossed we get some good news soon!

[Enough]

Ontrack is one of the best -- I would send it there. Also fuck that guy that did that - hope he/she was arrested?

It's a pretty fubar situation that is out of my hands. I have no idea what consequences might result as this portion is now above my pay grade, but I agree it's maddening to think there might not be any charges.

[Enough]

We just heard back from DriveSavers and they were able to recover they estimate 85-90%+ of the data deleted off the SAN, it's a Christmas miracle! $22k for the job, we are hopeful that insurance will cover. The VMs were able to load in their environment for our lost servers as well. Fingers crossed this will mostly make us whole, happy days!

[Isgrimnur]

Excellent news!

[Paingod]

Bricks were shat, fingers gnawed. I'm glad it came back with such a high success rate.

I've never had to do a pro recovery before, but have had abysmal luck with software recovery apps. I do my best to keep my backups happy, but you never really know when things will go completely sideways.

[Tao]

We have a pretty robust backup system/plan, with hourly snapshots and digital backup to our private cloud, we have often discussed just phasing out our tape backups but even the minimal chance of insider threat keeps them in place. All digital backup is just to susceptible. Glad to hear things turned out well for you and your organization.

[The Meal]

Wow, that's fantastic. Congrats on the reason for optimism!

[Paingod]

digital backup to our private cloud

'Round here that's a fancy way of saying "the external drive connected to the receptionist's desk in the other location" ...

[FishPants]

We have a pretty robust backup system/plan, with hourly snapshots and digital backup to our private cloud, we have often discussed just phasing out our tape backups but even the minimal chance of insider threat keeps them in place. All digital backup is just to susceptible. Glad to hear things turned out well for you and your organization.

If you go with online backups, at least make a couple of copies - one of which is made with an account that can only append, not delete/overwrite. That way Ransomware can't migrate into your backups (that shit keeps me awake at night).

[UsulofDoom]

How much Data needed to be stored? Terabytes ,Petabytes or more?

[Tao]

We have a pretty robust backup system/plan, with hourly snapshots and digital backup to our private cloud, we have often discussed just phasing out our tape backups but even the minimal chance of insider threat keeps them in place. All digital backup is just to susceptible. Glad to hear things turned out well for you and your organization.

If you go with online backups, at least make a couple of copies - one of which is made with an account that can only append, not delete/overwrite. That way Ransomware can't migrate into your backups (that shit keeps me awake at night).

So by private cloud I mean it's all internal/on-prem. We have multiple data centers with separate SANS and we preform cross-site backups which are kept in our network storage so there are always at least two copies of our backups, one at the local site, and one at an alternate site, plus we do snapshots and backup to tape which are then swapped between physical sites. We could lose an entire site and be operational again in a few hours. I suppose one of our system admins could try and ransom the data but it wouldn't take long to figure out who it was which sort of defeats the purpose. My bigger fear would be a disgruntled admin decides to start wiping data, hence why we still maintain physical copies on tape with limited access.

[Enough]

How much Data needed to be stored? Terabytes ,Petabytes or more?

Just under 12 TB, but lots of files, millions and millions of tiny db files, etc.

[FishPants]

We have a pretty robust backup system/plan, with hourly snapshots and digital backup to our private cloud, we have often discussed just phasing out our tape backups but even the minimal chance of insider threat keeps them in place. All digital backup is just to susceptible. Glad to hear things turned out well for you and your organization.

If you go with online backups, at least make a couple of copies - one of which is made with an account that can only append, not delete/overwrite. That way Ransomware can't migrate into your backups (that shit keeps me awake at night).

So by private cloud I mean it's all internal/on-prem. We have multiple data centers with separate SANS and we preform cross-site backups which are kept in our network storage so there are always at least two copies of our backups, one at the local site, and one at an alternate site, plus we do snapshots and backup to tape which are then swapped between physical sites. We could lose an entire site and be operational again in a few hours. I suppose one of our system admins could try and ransom the data but it wouldn't take long to figure out who it was which sort of defeats the purpose. My bigger fear would be a disgruntled admin decides to start wiping data, hence why we still maintain physical copies on tape with limited access.

But what permissions do those backup service accounts have on the target SAN? Are they only allowed to append, or can they do a full write/delete? Append shouldn't affect your backup process at all -- but it certainly blocks ransomware from spreading into your previously good online SAN backups.

If you haven't been hit by a ransomware outbreak, that's a good thing - but prep now so you can weather the inevitable storm (do some app whitelisting on at least one domain controller to prevent you losing domain access in the middle of an outbreak for example), and hopefully you've got a solid next gen endpoint security product with EDR. It's ugly out there!

[Punisher]

Speaking from experience Ransomware attacks are one of the most entertaining things you will ever have to deal with in IT. Nothing but good natured fun all around and can be fully resolved within an hour while you are at a party with your other IT staff and no loss of sleep at all. It's just real good fun!
It is also VERY good at letting you know failures in your data chain and backups so it really is just a good thing trying to help you out!

Also, if the data is real important to you (which this sound like) There is no limit to how many backups you can have. I've had clients that had 2 different on-site storage, 2 off-site where they take extra copies either daily or weekly and bring them to different peoples homes (pending security concerns obviously), 2 different cloud solutions, and a monthly archive in a bank box. Sorry.. This was AFTER they got hit with ransomware and almost had toshut the company down for good cause it hit all local and network data and killed the backups because they weren't configured correctly. They didn't have a cloud option at the time. They got REALLY lucky as this turned out to be one of the "honest" ransomers and they DID give the unlock info after paying $10k to get it. (We have also had some clients come to us and pay the ransom but never received the unlock info. Those ransomers REALLY suck. If you are going to blackmail me at least have the decency to uphold your end of the bargain. (and a few of those clients just went belly up due to the incident.))
For a LOT of companies backups are like IT in general. When everything is working they question why they have IT. Hey we haven't lost any data in the 20 years we have been open so that means we really don't need to go crazy with backups since the correlation is that we wont have to worry for at least another 20 years.
Yeah, I just don't get some clients.... I usually try to get them to understand by asking what would happen if I just walked into the server room and deleted all your data and your cheesy backup and then overwrote it just to make things fun. If they say they'd have to shut down, I'd ask how much is the company worth and then tell them they should probably have something in place to help protect that investment.
(My favorite clients are the ones that just trust us (me and the various companies I've worked for). You pay us to be the experts, please listen when we tell you to do somethnig.