Fundraising complete, next renewal is August 2022. Paypal Donation Links US dollars CDN Dollars

Emergency SAN Data Recovery Advice

Everything else!

Moderators: EvilHomer3k, Bakhtosh

Post Reply
User avatar
Enough
Posts: 14475
Joined: Tue Oct 12, 2004 11:05 pm
Location: Serendipity
Contact:

Emergency SAN Data Recovery Advice

Post by Enough »

Let's say that there was a really bad icky no damn good IT incident that results from when folks go bad. The impact was severe risk of data loss and said data is not just commercial widget data, but really valuable research that besides representing $50 million+ in research, but sadly is also irreplaceable loss for society in the greater good sense. Now let's continue this very dark fantasy. What if you had a RAID 5 SAN that a rogue deleted all the vdisks on as part of that nuclear strike and you were tasked with working miracles. Would you go with DriveSavers or Ontrack? Yes we have quotes from both ($20k+ but it's a no-brainer), but what say you OO? May the force be with you.
My blog (mostly photos): Fort Ephemera - My Flickr Photostream

“You only get one sunrise and one sunset a day, and you only get so many days on the planet. A good photographer does the math and doesn’t waste either.” ―Galen Rowell

Jeff V
Posts: 33190
Joined: Fri Oct 15, 2004 7:17 pm
Location: Nowhere you want to be.

Re: Emergency SAN Data Recovery Advice

Post by Jeff V »

When something similar happened years ago on a company server in Fairfax, VA, the case was investigated by the Secret Service (they were the cyber crime authority in the DC area). Had I not already taken expensive steps at restoring the data as best we could, they said they could have arranged recovery using non-commercially available tools. If you have any competent law enforcement investigating, they might have suggestions.
Black Lives Matter

User avatar
LordMortis
Posts: 63305
Joined: Tue Oct 12, 2004 11:26 pm

Re: Emergency SAN Data Recovery Advice

Post by LordMortis »

Call the Fed was my first thought as well. That's a straight to legal question. I'm going to have nightmares tonight.

User avatar
naednek
Posts: 9376
Joined: Tue Oct 19, 2004 9:23 pm
naednek’s avatar
Loading…

Re: Emergency SAN Data Recovery Advice

Post by naednek »

this data wasn't backed up elsewhere?
hepcat - "I agree with Naednek"

User avatar
Enough
Posts: 14475
Joined: Tue Oct 12, 2004 11:05 pm
Location: Serendipity
Contact:

Re: Emergency SAN Data Recovery Advice

Post by Enough »

naednek wrote:
Fri Sep 04, 2020 12:54 pm
this data wasn't backed up elsewhere?
Of course it was, but being a nonprofit my insistence that we have a third backup for proper 321 was never adequately funded and so backups were relying on a single Veritas Backup Exec backup and it was part of the incident. Thankfully, after nearly no sleep for a few days last Fri-Sun I was able to resurrect some, but the best I can do is back to April for the network drive shares and thankfully I can get early Aug for the web, db and GIS servers (this also took out 90% of our servers, luckily the DC, backups and file servers survived mostly). Make no mistake this was an attempt to nuke us from orbit, but we are not going to let that happen. I do have the entire campus IT community pitching in and actually have not touched the SAN since it got nuked to increase chances of professional data recovery. And the rest of the management team is certainly now onboard to fund the third backup and other IT enhancements we've been needing. I am taking full advantage of this to make sure we will end up stronger and more resilient in IT than we ever have been before.

Very few labs around the country can do the SAN recovery but the two I listed came recommended from a colleague at VMWare who has used both a number of times for similar issues successfully. The key is no writes and I think (fingers crossed) we are good there. We won't know until we can send in the drives to be evaluated in the lab but I can't do that until the insurance and police reports are complete. Then it's likely a couple of weeks to a month before we get that back. I just wanted to check hoping the OO Hive Mind may have faced a similar issue before and had opinions on DriveSavers and Ontrack/Kroll or other options. I have also of course scoured the web for said opinions. Thanks all!
My blog (mostly photos): Fort Ephemera - My Flickr Photostream

“You only get one sunrise and one sunset a day, and you only get so many days on the planet. A good photographer does the math and doesn’t waste either.” ―Galen Rowell

User avatar
gbasden
Posts: 6340
Joined: Wed Oct 13, 2004 1:57 am
Location: Sacramento, CA

Re: Emergency SAN Data Recovery Advice

Post by gbasden »

I had a very good experience in a similar situation with Ontrack, but it was about 7 or 8 years ago. I'm not sure how applicable that is at this point.

User avatar
FishPants
Server WhOOre
Posts: 4420
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada
fishpants’s avatar
Loading…

Re: Emergency SAN Data Recovery Advice

Post by FishPants »

Ontrack is one of the best -- I would send it there. Also fuck that guy that did that - hope he/she was arrested?
No.

User avatar
Enough
Posts: 14475
Joined: Tue Oct 12, 2004 11:05 pm
Location: Serendipity
Contact:

Re: Emergency SAN Data Recovery Advice

Post by Enough »

Thanks all, the team decided to go with DriveSavers and they got the drives on Friday for evaluation. I spoke to a fairly heavy user of both DriveSavers and OnTrack from VMWare who said he has had equally good luck with both and it looked like they would have a quicker turnaround. Fingers crossed we get some good news soon!
My blog (mostly photos): Fort Ephemera - My Flickr Photostream

“You only get one sunrise and one sunset a day, and you only get so many days on the planet. A good photographer does the math and doesn’t waste either.” ―Galen Rowell

User avatar
Enough
Posts: 14475
Joined: Tue Oct 12, 2004 11:05 pm
Location: Serendipity
Contact:

Re: Emergency SAN Data Recovery Advice

Post by Enough »

FishPants wrote:
Thu Sep 10, 2020 3:21 pm
Ontrack is one of the best -- I would send it there. Also fuck that guy that did that - hope he/she was arrested?
It's a pretty fubar situation that is out of my hands. I have no idea what consequences might result as this portion is now above my pay grade, but I agree it's maddening to think there might not be any charges.
My blog (mostly photos): Fort Ephemera - My Flickr Photostream

“You only get one sunrise and one sunset a day, and you only get so many days on the planet. A good photographer does the math and doesn’t waste either.” ―Galen Rowell

Post Reply