Problems since the recovery? Post them here please

Post by **FishPants** » Mon Sep 02, 2013 11:09 am

Let me know the issues (and there will be issues, from performance, broken links, and bad juju) here. I'm working through the stuff as I come across it (I anticipate e-mail isn't working right at the moment for example). I'll update here as issues are also fixed.

So far I've got to look into:

1. External Mailing
2. OOWiki
3. Deadlock SQL errors
4. Teamspeak server
5. Ventrilo Server
6. Set up SQL backups twice weekly (full DB) to new hosting provider FTP Server
7. Set up SQL backups to also move to Spideroak (Now Amazon S3, Weekly)
8. Set up monthly HTML directory backups to FTP
9. Set up monthly HTML directory backups to SpiderOak (Now Amazon S3, Weekly)

Updates:

- OOWiki I think is fixed. At least it doesn't barf out errors all the time.
- External email worked out of the gate (how about that!)
- Teamspeak is online, new password posted in the first post of the gaming services area in PC gaming.
- More smilies added (not something that was broken, but requested)
- Backups - Currently backing up the entire server nightly (Full backup) to a local file + OVH Hosted FTP backup server. Retaining backups for 60 days on the FTP server, so we can roll back as necessary. No more incremental backup problems/bullshit.
- Backups now being placed in reduced redundancy storage at S3 on a weekly basis.

JetFred · Post by **JetFred** » Mon Sep 02, 2013 11:45 am

Testing from Tapatalk because you love it so much. You are the hero of Labor Day, fish man.

Post by **FishPants** » Mon Sep 02, 2013 11:58 am

JetFred wrote:Testing from Tapatalk because you love it so much. You are the hero of Labor Day, fish man.

ARGH!! I forgot about Tapatalk and that upgrade.. That happened after the last good backup in July.. Son.. Of.. A..

JetFred · Post by **JetFred** » Mon Sep 02, 2013 12:04 pm

I came here to praise you and it seems like I'm being an obsessive pain, because I'm about to post an error. But realize I have nothing but patience and respect for you in this rebuild process. I got on Steam to send you a thank you game but saw you haven't been online for like 4 years. Offer still stands, though.

Anyway, I got this error trying to post in the Pay It Forward thread in the trade forum:

General Error

SQL ERROR [ mysqli ]

Deadlock found when trying to get lock; try restarting transaction [1213]

An SQL error occurred while fetching this page. Please contact the Board Administrator if this problem persists.

Post by **FishPants** » Mon Sep 02, 2013 12:20 pm

JetFred wrote:I came here to praise you and it seems like I'm being an obsessive pain, because I'm about to post an error. But realize I have nothing but patience and respect for you in this rebuild process. I got on Steam to send you a thank you game but saw you haven't been online for like 4 years. Offer still stands, though.

Anyway, I got this error trying to post in the Pay It Forward thread in the trade forum:

General Error

SQL ERROR [ mysqli ]

Deadlock found when trying to get lock; try restarting transaction [1213]

An SQL error occurred while fetching this page. Please contact the Board Administrator if this problem persists.

Interesting error... I'm pruning the user database again and if I am doing too many transactions at once I get a 500 error -- wondering if the two are related (you are trying to post while I'm thrashing the hell out of it). Let me look into this a bit; I might have the wrong lock type in the config file.

That's the major drag, sure losing a few weeks of posts sucks -- but losing my SQL config file that was years of tuning... that blew. I'm starting from a vanilla config -- sucks!

I backed out a config entry that I think may have allowed for deadlocks.. Please let me know if you see this again (I just saw it myself)

JetFred · Post by **JetFred** » Mon Sep 02, 2013 12:31 pm

To further tax your work, my son just registered. That registration succeeded against my expectations. He was able to post in that same thread, but when I tried to post there and here, I would get that error. Presumably if this is showing up now, I got it to post successfully eventually.

Post by **FishPants** » Mon Sep 02, 2013 12:39 pm

JetFred wrote:To further tax your work, my son just registered. That registration succeeded against my expectations. He was able to post in that same thread, but when I tried to post there and here, I would get that error. Presumably if this is showing up now, I got it to post successfully eventually.

Yep it seems when I'm beating up the database through pruning that it was throwing up those errors elsewhere.. That's odd for InnoDB.. will have to read up on what causes that.

KKBlue · Post by **KKBlue** » Mon Sep 02, 2013 1:27 pm

Fishy (cute nic name for a moment) smiles are necessary for sure!

I had gotten a few error messages as I was all over the forum like a dog seeing it's master for the first time in a very long time. I didn't write (copy) any of the messages nor can I remember what I was doing when it happened (dog gone nutzo). The error didn't cause me not to do something if I pressed the button again.

My point Fishy, is, like you had mentioned, it's going to take some time before OO is at a full recovery. My thought when the errors were coming up for me, it was because you were doing things on your end. Plus we are all running back here, all love blind and attention starved folk. Keep on doing what you are doing and know your efforts are appreciated. PLUS we (talking for the consensus without permission) do want you to take some time for yourself today. Please don't be in front of the computer all day. We will do that for you, ha!

(got an error message when I hit send, so people remember to save your posts first, just in case. Ended up submitted after the 5th try over the corse of about 5 minutes. Again, my guess is you were banging around and the "computer term I'm unfamiliar with" just couldn't keep up)
General Error
SQL ERROR [ mysqli ]
Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) [2002]
An sql error occurred while fetching this page. Please contact an administrator if this problem persists.

Post by **FishPants** » Mon Sep 02, 2013 1:46 pm

Yeah been trying to figure out the deadlock issue when I have a large Innodb buffer size.. Learned the hard way that Centos (even 6.x) only uses MySQL v5.1.. That's a bit of a drag since there's some further performance enhancements in 5.5.4+ that I would like to use (to separate out the large innodb buffer into 1 gig chunks) that aren't currently available to me.

Anyways it only seems to be if I am running large operations on the user table etc that it results in deadlocks. I'm going to stop doing that for the time being, please report any further deadlocks in here please -- would like to know if it's happening outside of me doing that.

JetFred · Post by **JetFred** » Mon Sep 02, 2013 3:54 pm

Are unread post links jumping to the first unread post like they should?

Post by **FishPants** » Mon Sep 02, 2013 4:04 pm

JetFred wrote:Are unread post links jumping to the first unread post like they should?

They are for me, the only time that doesn't apply is with the global stickied subjects (Like the Welcome Back message that's stickied in all forums).

JetFred · Post by **JetFred** » Mon Sep 02, 2013 4:18 pm

FishPants wrote:
JetFred wrote:Are unread post links jumping to the first unread post like they should?
They are for me, the only time that doesn't apply is with the global stickied subjects (Like the Welcome Back message that's stickied in all forums).

Ah, well it's a good chance that's the only one I was noticing it on.

Carry on. Oh, and way to go above and beyond with the smilies.

wonderpug · Post by **wonderpug** » Mon Sep 02, 2013 4:31 pm

I can't seem to find the series of threads I made about Monty Python references just before the server went down.

Post by **FishPants** » Mon Sep 02, 2013 4:32 pm

wonderpug wrote:I can't seem to find the series of threads I made about Monty Python references just before the server went down.

The last DB backup I had was from July 21st, so if you made it after that unfortunately we didn't have access to that data.

Pyperkub · Post by **Pyperkub** » Mon Sep 02, 2013 5:21 pm

Tapatalk 4 keeps throwing an error that the topic doesn't exist whenever I try to read the welcome back please read topic.

That's the only problem I've seen yet...

wonderpug · Post by **wonderpug** » Mon Sep 02, 2013 6:17 pm

FishPants wrote:
wonderpug wrote:I can't seem to find the series of threads I made about Monty Python references just before the server went down.
The last DB backup I had was from July 21st, so if you made it after that unfortunately we didn't have access to that data.

Probably the wrong thread for the joke, but I was trying to insinuate that it was the mother of all Monty Python references that caused the disaster.

Blackhawk · Post by **Blackhawk** » Mon Sep 02, 2013 6:51 pm

JetFred wrote:Are unread post links jumping to the first unread post like they should?

Not for me. They are landing me all over whichever thread I'm reading, either sending me back half a page, or to the next page - anywhere but the next unread post.

JetFred · Post by **JetFred** » Mon Sep 02, 2013 7:20 pm

wonderpug wrote:
FishPants wrote:
wonderpug wrote:I can't seem to find the series of threads I made about Monty Python references just before the server went down.
The last DB backup I had was from July 21st, so if you made it after that unfortunately we didn't have access to that data.
Probably the wrong thread for the joke, but I was trying to insinuate that it was the mother of all Monty Python references that caused the disaster.

I see now. It's like if I had said that just before the server went down, I had asked for the third time whether anyone still keeps an eye on Derek Smart's career.

naednek · Post by **naednek** » Tue Sep 03, 2013 12:32 am

Whenever I post, it will post it twice...

Watch...

naednek · Post by **naednek** » Tue Sep 03, 2013 12:33 am

Whenever I post, it will post it twice...

Watch...

naednek · Post by **naednek** » Tue Sep 03, 2013 12:35 am

Ok... I think I found the issue.

In the past (pre loss of data) When I posted something, I always the back button to go back to the forum or post.... No problems

Now when I do the same thing, I hit the back button, and it reposts my message

Not sure what changed.

Post by **FishPants** » Tue Sep 03, 2013 9:25 am

naednek wrote:Ok... I think I found the issue.

In the past (pre loss of data) When I posted something, I always the back button to go back to the forum or post.... No problems

Now when I do the same thing, I hit the back button, and it reposts my message

Not sure what changed.

I've been trying to replicate this problem without any luck.. Can you please clear your browser cache etc and log out of OO then back in please? Wondering if caching from the old site is conflicting with the new site (post orders and IDs, and so on).

hentzau · Post by **hentzau** » Tue Sep 03, 2013 9:45 am

FishPants wrote:
naednek wrote:Ok... I think I found the issue.

In the past (pre loss of data) When I posted something, I always the back button to go back to the forum or post.... No problems

Now when I do the same thing, I hit the back button, and it reposts my message

Not sure what changed.
I've been trying to replicate this problem without any luck.. Can you please clear your browser cache etc and log out of OO then back in please? Wondering if caching from the old site is conflicting with the new site (post orders and IDs, and so on).

I was having this problem before the outage, but I have just tried to modify my posting behavior. And I would get the behavior in Safari and Chrome. I'll test it right now after I post this.

hentzau · Post by **hentzau** » Tue Sep 03, 2013 9:45 am

FishPants wrote:
naednek wrote:Ok... I think I found the issue.

In the past (pre loss of data) When I posted something, I always the back button to go back to the forum or post.... No problems

Now when I do the same thing, I hit the back button, and it reposts my message

Not sure what changed.
I've been trying to replicate this problem without any luck.. Can you please clear your browser cache etc and log out of OO then back in please? Wondering if caching from the old site is conflicting with the new site (post orders and IDs, and so on).

I was having this problem before the outage, but I have just tried to modify my posting behavior. And I would get the behavior in Safari and Chrome. I'll test it right now after I post this.

hentzau · Post by **hentzau** » Tue Sep 03, 2013 9:47 am

Yep. Make my post, let it complete, and then go back a few pages and bingo! Dup post. But like I said, it was working like that before the server went tits up.

naednek · Post by **naednek** » Tue Sep 03, 2013 10:19 am

hentzau wrote:Yep. Make my post, let it complete, and then go back a few pages and bingo! Dup post. But like I said, it was working like that before the server went tits up.

Weird. I use chrome and it never did to me until yesterday. I'll try what fishpants said the next time I reboot my PC.

Holman · Post by **Holman** » Tue Sep 03, 2013 11:20 am

I have no OO problems--I just wanted to throw out a big thank you for the recovery effort!

Zaxxon · Post by **Zaxxon** » Tue Sep 03, 2013 11:45 am

Some threads are unreadable in Tapatalk. One example is the 'Welcome back please read' thread. Tapatalk shows it in the thread list but errors out with thread does not exist when it's tapped.

LordMortis · Post by **LordMortis** » Tue Sep 03, 2013 11:47 am

wonderpug wrote:
FishPants wrote:
wonderpug wrote:I can't seem to find the series of threads I made about Monty Python references just before the server went down.
The last DB backup I had was from July 21st, so if you made it after that unfortunately we didn't have access to that data.
Probably the wrong thread for the joke, but I was trying to insinuate that it was the mother of all Monty Python references that caused the disaster.

I made a "Manning would have been cooler if he wanted to be called Loretta" crack on FB when it broke the news but I swear to Pancake OO was already crashed.

Post by **FishPants** » Tue Sep 03, 2013 12:41 pm

Zaxxon wrote:Some threads are unreadable in Tapatalk. One example is the 'Welcome back please read' thread. Tapatalk shows it in the thread list but errors out with thread does not exist when it's tapped.

I'm checking on my iPad, and no issues with Tapatalk at all yet on my Android it says the topic does not exist. Odd.

Post by **FishPants** » Tue Sep 03, 2013 12:58 pm

Zaxxon wrote:Some threads are unreadable in Tapatalk. One example is the 'Welcome back please read' thread. Tapatalk shows it in the thread list but errors out with thread does not exist when it's tapped.

Just updated the server side Tapatalk plugin; that should be working now yes?

Zaxxon · Post by **Zaxxon** » Tue Sep 03, 2013 1:24 pm

FishPants wrote:
Zaxxon wrote:Some threads are unreadable in Tapatalk. One example is the 'Welcome back please read' thread. Tapatalk shows it in the thread list but errors out with thread does not exist when it's tapped.
Just updated the server side Tapatalk plugin; that should be working now yes?

Confirmed. Thanks!

silverjon · Post by **silverjon** » Tue Sep 03, 2013 5:32 pm

Blackhawk wrote:
JetFred wrote:Are unread post links jumping to the first unread post like they should?
Not for me. They are landing me all over whichever thread I'm reading, either sending me back half a page, or to the next page - anywhere but the next unread post.

What seems to be happening is it sends you to the first post that would have been unread by you at the time of the restored backup. Or at least that's what it looks like to me. Just like how there are threads marked unread by me that were last posted to in July. I would have marked those as read, but sometime after the backup.

Blackhawk · Post by **Blackhawk** » Tue Sep 03, 2013 10:41 pm

silverjon wrote:
Blackhawk wrote:
JetFred wrote:Are unread post links jumping to the first unread post like they should?
Not for me. They are landing me all over whichever thread I'm reading, either sending me back half a page, or to the next page - anywhere but the next unread post.
What seems to be happening is it sends you to the first post that would have been unread by you at the time of the restored backup. Or at least that's what it looks like to me. Just like how there are threads marked unread by me that were last posted to in July. I would have marked those as read, but sometime after the backup.

I don't think that's it. Let's say there is a thread with three pages, and I've read half of page 3, then leave. When I come back and see that it is now four pages and hit 'next unread', it takes me to page four, not the middle of 3. This happens even on threads created since the reboot.

On a page that is 1/4 read but is up the 3/4 done, when I click 'next unread', I am sent about 1/2 of the way into the page.

I've cleared my cache and manually deleted my OO cookies, but it keeps happening.

silverjon · Post by **silverjon** » Wed Sep 04, 2013 3:00 am

Then in my highly technical opinion, yours is being weird.

Post by **FishPants** » Wed Sep 04, 2013 7:06 am

silverjon wrote:Then in my highly technical opinion, yours is being weird.

I have to agree here, I'm not seeing this behaviour on my end.

Anyone else with the same problem?

Zaxxon · Post by **Zaxxon** » Wed Sep 04, 2013 9:05 am

'First unread' is working fine here, in Chrome, IE and Tapatalk.

Lassr · Post by **Lassr** » Tue Sep 10, 2013 10:31 am

past few days I hang on the screen "Your Message has been posted successfully" it just churns forever until I hit see my message or return to last forum topic or whatever it says.

I also notice that the OO is constantly trying to load something. My tab in Firefox shows the little circle rotating on my OO tab as if it never stops loading.

LordMortis · Post by **LordMortis** » Tue Sep 10, 2013 10:36 am

Lassr wrote:past few days I hang on the screen "Your Message has been posted successfully" it just churns forever until I hit see my message or return to last forum topic or whatever it says.

I also notice that the OO is constantly trying to load something. My tab in Firefox shows the little circle rotating on my OO tab as if it never stops loading.

I had the same problem beginning yesterday. Clearing my history fixed it.

Post by **FishPants** » Tue Sep 10, 2013 10:37 am

Lassr wrote:past few days I hang on the screen "Your Message has been posted successfully" it just churns forever until I hit see my message or return to last forum topic or whatever it says.

I also notice that the OO is constantly trying to load something. My tab in Firefox shows the little circle rotating on my OO tab as if it never stops loading.

Why the heck am I not experiencing this? Between the weird double posting behaviour and few people talking about something stuck loading -- not sure what the deal is. If anyone has a diagnostic firefox plugin and can paste some of the content while the issue is happening that would *really* help.