So that sucked.

Discuss site matters here

Moderators: FishPants, ooRip

User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

So that sucked.

Post by FishPants »

So here's some learning points:

1. Rip was right. So very right. I feel shame.
2. Database won't convert, I'll open a ticket.. We stay on the old version.
3. First restore took about 30 hours.. Then I screwed it up again (whoops, wrong script).
4. I made some tweaks (!) to the config, and the next restore took exactly 15 minutes.
5. No that's not a joke.. 15 minutes.

Please report any funkiness. I'm going to go feel shame with a glass of red wine.

:obscene-drinkingdrunk:
No.
User avatar
Bad Demographic
Posts: 7772
Joined: Wed Oct 13, 2004 9:21 am
Location: Las Cruces, NM

Re: So that sucked.

Post by Bad Demographic »

No funkiness, but I had to look in the Books Read thread to figure out the correct spelling of my username - and I had to guess at my password, too. :oops:
(I've written them down now - for the future)
"Those who can make you believe absurdities can make you commit atrocities." Voltaire

Black Lives Matter
User avatar
Zaxxon
Forum Moderator
Posts: 28118
Joined: Wed Oct 13, 2004 12:11 am
Location: Surrounded by Mountains

Re: So that sucked.

Post by Zaxxon »

No shame. We're back, we appear to be whole, and you live to fight another day. Lessons learned, and all that.
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

On the plus side the entire database is in RAM now, and the "write" threads is up to 16 -- so that means restores are WAY faster.

At least now I can reload the backup to another DB and test over there, even on a clean install the phpbb3 script to upgrade the database has a weird failure.. I may need to update our MySQL version to a more modern version. That's for another day.
No.
User avatar
gbasden
Posts: 7664
Joined: Wed Oct 13, 2004 1:57 am
Location: Sacramento, CA

Re: So that sucked.

Post by gbasden »

No shame at all! Thanks so much for all the effort you put in!
User avatar
Kraken
Posts: 43688
Joined: Tue Oct 12, 2004 11:59 pm
Location: The Hub of the Universe
Contact:

Re: So that sucked.

Post by Kraken »

Yay, normality is restored, and 10% better than it was before.
User avatar
Smoove_B
Posts: 54567
Joined: Wed Oct 13, 2004 12:58 am
Location: Kaer Morhen

Re: So that sucked.

Post by Smoove_B »

Have you patented the methodology that you used to get from 30 hours to 15 minutes, because that seems astoundingly more efficient?

Thanks for all your hard work. Have a drink on me!
Maybe next year, maybe no go
User avatar
Exodor
Posts: 17196
Joined: Tue Oct 12, 2004 11:10 pm
Location: Portland, OR

Re: So that sucked.

Post by Exodor »

gbasden wrote:No shame at all! Thanks so much for all the effort you put in!
Speak for yourselves.

Image

:P


Thanks for all your hard work.
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

Image
No.
User avatar
LawBeefaroni
Forum Moderator
Posts: 55316
Joined: Fri Oct 15, 2004 3:08 pm
Location: Urbs in Horto, outrageous taxes on everything

Re: So that sucked.

Post by LawBeefaroni »

Enjoy the well deserved wine.
" Hey OP, listen to my advice alright." -Tha General
"No scientific discovery is named after its original discoverer." -Stigler's Law of Eponymy, discovered by Robert K. Merton

MYT
User avatar
cheeba
Posts: 8727
Joined: Tue Jan 04, 2005 3:32 am

Re: So that sucked.

Post by cheeba »

As always, thanks for all you guys do with the site! I especially appreciate the updated page and not just coming to a 404 or something.
User avatar
Carpet_pissr
Posts: 19978
Joined: Thu Nov 04, 2004 5:32 pm
Location: Columbia, SC

Re: So that sucked.

Post by Carpet_pissr »

Smoove_B wrote:Have you patented the methodology that you used to get from 30 hours to 15 minutes, because that seems astoundingly more efficient?
Unpossible. Pied Piper already holds patents on the middle out compression solution that saved us all from internet purgatory.
User avatar
Daehawk
Posts: 63524
Joined: Sat Jan 01, 2005 1:11 am

Re: So that sucked.

Post by Daehawk »

Bad Demographic wrote:No funkiness, but I had to look in the Books Read thread to figure out the correct spelling of my username - and I had to guess at my password, too. :oops:
(I've written them down now - for the future)
Strange as Im still logged in.

Fish thank you so much for it all. Sorry it went badly. umm better luck next time? :)
--------------------------------------------
I am Dyslexic of Borg, prepare to have your ass laminated.
I guess Ray Butts has ate his last pancake.
http://steamcommunity.com/id/daehawk
"Has high IQ. Refuses to apply it"
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

Ehh it wasn't that stressful to be honest.. I had solid backups, it didn't involve a lot of my time other than check it once in awhile and update the HTML file for status.. I'm just really pissed off that the performance tweaks I found were something I could have used 10 years ago -- and it was through a forum post that referred to an academic that posted some crazy whitepaper, and a link to a TL;DR version of it with 5 settings for InnoDB tables. I set those (based on RAM) and viola -- fast restore. Crap default values, and obscure articles on how to tune to something reasonable -- frustration!

Anyways now I need to figure out how to create a mirror site, restore the DB there and test the upgrade script there.. I'm wondering if people hitting the site while it was upgrading was the problem (I didn't turn apache off, so I could serve up the status page..).
No.
User avatar
Daehawk
Posts: 63524
Joined: Sat Jan 01, 2005 1:11 am

Re: So that sucked.

Post by Daehawk »

I only checked every 5 minutes so it wasn't me. :P
--------------------------------------------
I am Dyslexic of Borg, prepare to have your ass laminated.
I guess Ray Butts has ate his last pancake.
http://steamcommunity.com/id/daehawk
"Has high IQ. Refuses to apply it"
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

Daehawk wrote:I only checked every 5 minutes so it wasn't me. :P
Image
No.
User avatar
dbt1949
Posts: 25687
Joined: Wed Oct 13, 2004 12:34 am
Location: Hogeye Arkansas

Re: So that sucked.

Post by dbt1949 »

Without OO I don't think I need the internet. Just boring Trump crap.
Ye Olde Farte
Double Ought Forty
aka dbt1949
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

You know we have a core group of people that love this place when Smoove is logged in before I am.. and I'm the guy bringing the site back online.. "Oh look the restore is finished.. let me change this config file.. ok done.. ok lets restart the web server and login.. hey there's smoove!".

Not to mention another 67 "guest users" (we seemed to dump most cached login information on the restore). Sickos! The lot of ya!
No.
User avatar
Isgrimnur
Posts: 82085
Joined: Sun Oct 15, 2006 12:29 am
Location: Chookity pok
Contact:

Re: So that sucked.

Post by Isgrimnur »

What do you expect me to do all day? Work?!
It's almost as if people are the problem.
User avatar
Blackhawk
Posts: 43490
Joined: Tue Oct 12, 2004 9:48 pm
Location: Southwest Indiana

Re: So that sucked.

Post by Blackhawk »

cheeba wrote:As always, thanks for all you guys do with the site! I especially appreciate the updated page and not just coming to a 404 or something.
Or being told you're banned for being a spammer! :o
(˙pǝsɹǝʌǝɹ uǝǝq sɐɥ ʎʇıʌɐɹƃ ʃɐuosɹǝd ʎW)
User avatar
Zarathud
Posts: 16433
Joined: Fri Oct 15, 2004 10:29 pm
Location: Chicago, Illinois

Re: So that sucked.

Post by Zarathud »

Very happy to see the harmony of OO restored to the Universe.

A forced break was good, but there were serious withdrawal symptoms.
"If the facts don't fit the theory, change the facts." - Albert Einstein
"I don't stand by anything." - Trump
“Bad men need nothing more to compass their ends, than that good men should look on and do nothing.” - John Stuart Mill, Inaugural Address Delivered to the University of St Andrews, 2/1/1867
“It is the impractical things in this tumultuous hell-scape of a world that matter most. A book, a name, chicken soup. They help us remember that, even in our darkest hour, life is still to be savored.” - Poe, Altered Carbon
User avatar
Daehawk
Posts: 63524
Joined: Sat Jan 01, 2005 1:11 am

Re: So that sucked.

Post by Daehawk »

Im feeling much better now.

Enlarge Image
--------------------------------------------
I am Dyslexic of Borg, prepare to have your ass laminated.
I guess Ray Butts has ate his last pancake.
http://steamcommunity.com/id/daehawk
"Has high IQ. Refuses to apply it"
User avatar
Archinerd
Posts: 6836
Joined: Fri Aug 25, 2006 11:18 am
Location: Shikaakwa

Re: So that sucked.

Post by Archinerd »

Hooray, tomorrow is going to feel like a vacation day at work now!
User avatar
Moat_Man
Posts: 1653
Joined: Fri Oct 15, 2004 3:44 am
Location: Burnaby, BC

Re: So that sucked.

Post by Moat_Man »

What happened to BlackPearl?
End of line
¯¯¯¯¯¯¯¯¯
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

Some interesting database issues with mods. All fixed, no more touching for tonight/this morning.
No.
User avatar
Moat_Man
Posts: 1653
Joined: Fri Oct 15, 2004 3:44 am
Location: Burnaby, BC

Re: So that sucked.

Post by Moat_Man »

FishPants wrote:Some interesting database issues with mods. All fixed, no more touching for tonight/this morning.
I'm back in Black. Time to hit sack.
End of line
¯¯¯¯¯¯¯¯¯
User avatar
Daehawk
Posts: 63524
Joined: Sat Jan 01, 2005 1:11 am

Re: So that sucked.

Post by Daehawk »

Hmm came back just now and it reset me to default silver style. Had to reset it to prosilver 2.
--------------------------------------------
I am Dyslexic of Borg, prepare to have your ass laminated.
I guess Ray Butts has ate his last pancake.
http://steamcommunity.com/id/daehawk
"Has high IQ. Refuses to apply it"
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

Yeah in short what I did was take the current DB, restore it to another DB (under a fictitious URL/domain).. I accessed it (by editing my hosts file) and ok great I have a mirror.. I read documentation on how to do the upgrade that's been failing (more forum posts).. Hey it says disable all styles and mods.. so I do.. then I realize I am using the same production config so I'm modifying this DB..

Whoops..

it's late.. my bad. restored some mod tables, seems to have fixed it.
No.
Jeff V
Posts: 36414
Joined: Fri Oct 15, 2004 7:17 pm
Location: Nowhere you want to be.

Re: So that sucked.

Post by Jeff V »

It seems that every time there's an extended OOutage, someone goes missing and Daehawk looks well-fed. Not that it's proof of anything, just an observation.
Black Lives Matter
User avatar
ImLawBoy
Forum Admin
Posts: 14950
Joined: Tue Oct 12, 2004 9:49 pm
Location: Chicago, IL
Contact:

Re: So that sucked.

Post by ImLawBoy »

FishPants wrote:All fixed, no more touching for tonight/this morning.
You're not the boss of me.
That's my purse! I don't know you!
User avatar
Rip
Posts: 26891
Joined: Tue Oct 12, 2004 9:34 pm
Location: Cajun Country!
Contact:

Re: So that sucked.

Post by Rip »

Enlarge Image
User avatar
Smoove_B
Posts: 54567
Joined: Wed Oct 13, 2004 12:58 am
Location: Kaer Morhen

Re: So that sucked.

Post by Smoove_B »

FishPants wrote:You know we have a core group of people that love this place when Smoove is logged in before I am.. and I'm the guy bringing the site back online.. "Oh look the restore is finished.. let me change this config file.. ok done.. ok lets restart the web server and login.. hey there's smoove!".
The real story here is who told me that it was up and running. :ninja:
Maybe next year, maybe no go
User avatar
Rip
Posts: 26891
Joined: Tue Oct 12, 2004 9:34 pm
Location: Cajun Country!
Contact:

Re: So that sucked.

Post by Rip »

Putin?
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

So the good news is that I was able to convert the database (after I stopped screwing up this one) without an error. Apparently having you clowns hammering the system while I was converting it was causing problems (I would have thought the conversion script would frigging lock the DB.. I guess not).

So after the holidays I will try again, but will have to disable the web server (so no status page). The good news is that it only took about an hour to do (and if it gets hosed, recovery is 30 mins). That sure beats a couple days of restore time!
No.
User avatar
Zaxxon
Forum Moderator
Posts: 28118
Joined: Wed Oct 13, 2004 12:11 am
Location: Surrounded by Mountains

Re: So that sucked.

Post by Zaxxon »

Excellent! Are there any user-facing benefits with 3.1?
User avatar
FishPants
Server WhOOre
Posts: 4658
Joined: Fri Oct 15, 2004 1:38 pm
Location: Canada

Re: So that sucked.

Post by FishPants »

Zaxxon wrote:Excellent! Are there any user-facing benefits with 3.1?
Take a look at our test instance :) I really like it, but it's different -- the masses may revolt!
No.
User avatar
Rip
Posts: 26891
Joined: Tue Oct 12, 2004 9:34 pm
Location: Cajun Country!
Contact:

Re: So that sucked.

Post by Rip »

The lack of something with a black background is certainly revolt worthy.

Has black done something wrong?
Last edited by Rip on Wed Nov 16, 2016 4:08 pm, edited 1 time in total.
User avatar
Blackhawk
Posts: 43490
Joined: Tue Oct 12, 2004 9:48 pm
Location: Southwest Indiana

Re: So that sucked.

Post by Blackhawk »

No. Badhawk.
(˙pǝsɹǝʌǝɹ uǝǝq sɐɥ ʎʇıʌɐɹƃ ʃɐuosɹǝd ʎW)
User avatar
DD*
Posts: 4706
Joined: Thu Nov 09, 2006 2:42 pm
Location: Detroit, where the weak are killed and eaten

Re: So that sucked.

Post by DD* »

Rip wrote:The lack of something with a black background is certainly revolt worthy.

Has black done something wrong?
Image
Are you a prostitute Rip? Because you blow the margins more than a $5 hooker. -rshetts2

Much like bravery is acting in spite of fear, being a functioning adult is acting responsibly in the face of temptation. -Isg
User avatar
stessier
Posts: 29816
Joined: Tue Dec 21, 2004 12:30 pm
Location: SC

Re: So that sucked.

Post by stessier »

Thank you for the frequent updates during the downtime. It made the withdrawals a lot easier to handle knowing someone was keeping tabs on the situation.
I require a reminder as to why raining arcane destruction is not an appropriate response to all of life's indignities. - Vaarsuvius
Global Steam Wishmaslist Tracking
Running____2014: 1300.55 miles____2015: 2036.13 miles____2016: 1012.75 miles____2017: 1105.82 miles____2018: 1318.91 miles__2019: 2000.00 miles
Post Reply