Resolved Explanation of the January 23rd Outage

[vtcnc]

Nelson was working on some of the legacy add-on issues that have been affecting performance and usability of the site. Nelson had deleted a folder that was from the previous version of the site software. This caused the site to crash.

Nelson then went to the web-host and ensured we had a backup. Our site currently get one monthly backup and daily backups. More on this in a minute when we talk about next steps. The backup didn't appear to have the attachments contained within, a total mystery to us, the web-host and the developers at Xenforo. By Friday night, we were pretty panicked.

Over the weekend, Nelson escalated the backup problem with management at our web-host. This got enough attention that they were willing to bring in the Xenforo developer into the ticket email chain so they could discuss the technical features of how the site software works, directory structure, backup scripts, etc. But all of this was not enough to locate the files attachments from January 23rd.

What exactly went wrong? Well, NO files, avatars, photos or thumbnails of photos were present. What mystified us is that some photos were loading but not all. We had to go back to January 1, the monthly backup to see what we could find there.

In the end, Xenforo found the directory of attachments and was able to restore the site to January 1. This is GREAT news in that we haven't lost the forum. It is BAD news in that anything new created between January 1 and January 23rd is likely to have been lost. For this, we are truly sorry. Following are the impacts to our members and what can be done to remedy the situation. Some things can be rectified by staff here at H-M, some cannot due to the nature of the crash and the timing:

• Threads/Posts - For those of you that posted new threads with photos between January 1-23, we do not have a backup. You will need to recreate this to the best of your ability.
• Account Upgrades - For those of you that upgraded your account and paid via PayPal, please post a response in the forum and I will respond directly to you via PM. The process here will be simple - you notify me that your upgrade was lost, I will verify with PayPal that we received a donation, then I will restore your upgrade to the level you were at prior to the crash. I will prioritize the requests for this category problem by user upgrade starting with Sponsor and commencing in the following order: Commercial, Diamond, Platinum, Gold, Silver, Registered.
• Photos/Albums - For those of you who uploaded photos and videos to the Gallery or your own personal album between January 1-23, we do not have a backup. You will need to upload those files again if you wish them to be there.
• Avatars - Same as photos/albums

There may be other problems that we have not covered here and have yet to emerge resulting from this crash. Please do not hesitate to post a question in this forum if you something comes up.

Preventative Actions (for the next time disaster strikes)

We have already increased the frequency of backups from monthly to weekly and daily. These will rotate over time so we will have far more coverage of backup data.
Nelson provided credentials for me to access the admin panel, support ticket system with the web-host and Xenforo developers so that I can create contingency plans, audit the site for weaknesses and put plans in place to prevent this sort of thing from happening again.

OK. We ask again that you post your questions and support requests here so we have a one stop shop for solving any residual problems from this crash as quickly as possible. We will answer them as soon and as completely as we can. Thanks everyone for your loyalty, patience and understanding.

 

[mickri]

Just to let you know that my thread on my HF welder has everything including photos up to last Wednesday, January 22.

Thanks so much for all of your hard work and everybody else who was involved to get the site back up and running.

 

[vtcnc]

Just to let you know that my thread on my HF welder has everything including photos up to last Wednesday, January 22.

Thanks so much for all of your hard work and everybody else who was involved to get the site back up and running.

Nelson and I were just going over a long part of the explanation from the web-host. It may turn out that they restored most of the photos. We will see over the next couple of days.

 

[LEEQ]

When you get a chance I upgraded to gold. I'm not to worried right this second. Thanks for your hard work

 

[DavidR8]

Thanks so much to everyone that worked so hard to get us right again. I so appreciate everyone here!

 

[Downunder Bob]

Thanks Bryan for the explanation, I had been wondering what was going on, but had no way to contact anyone to find out. I'm not aware as of today of any thing missing from my stuff, but if I find any I'll let you know. At least my avatar is back up again.

Thanks for all your help, hard work, and also thanks to Nelson, I thought he had retired from active duty. Good on him for getting back into the fray.

 

[Lo-Fi]

Thanks for the hard work getting it sorted.

I lost my supporter status having upgraded a couple of day before the crash, to add to the support queue please.

Thanks again!

 

[vtcnc]

Thanks everyone. I'm going to make this thread locked so if people need to post a support ticket - they will need to do so by following this link and posting a new thread there.