Sunday the 28th of August I was made aware by people that we were sending out SPAM. I checked this, but from the information provided to me (e-mail headers) I could not trace this back to the RPGWatch server. An RPGWatch account was listed as sender, but this can be spoofed. Still, I decided to block outgoing mail for the time being, did some other stuff as well and rebooted the server.
Looking back, I should have made a backup first, as the server became unreachable after that.
There are several ways to access the server, including a serial interface, but none of them worked. So calling customer support of the provider was the only option left. That took some time, phone calls and even Twitter interactions.
Let's just say that you do not need to be very skilled to work at customer support and you certainly do not need to listen to your customer. All you need to do is hear that it is problem X and my paper says that means solution Y.
Quite a bit of frustration later the provider found out that the motherboard was defect. I mentioned to them that I'm not interested in a working server as I already has an other server I was working to get operational. All I wanted was to have the data between the 17th and the 28th. Again there was no reason to listen to me, they replaced the motherboard and in the process also re-installed the default software (Linux, Apache, etc.) on the disk, removing all other data that was on it.
There was supposed to be a job running that did a daily backup to a separate backup server and that worked as I have regularly checked that, but for whatever reason the backup server had no data on it. I don't know why that is and neither does the provider.
So, I do not know if we were hacked and that whoever hacked us made all of this happen or that it was indeed a hardware failure (which still does not explain the issue with the backup server).
When playing the backup back, I also found out that there were some tables in the database that were not correct. These are the following:
- The thumbs-up's below the posts are gone. It works now again though.
- Everything related to the CRPG analyzer is gone and the analyzer does not work either
- The configuration on the home page, which allowed you to filter the posts does not work at the moment and you will have to renew your settings once it is operational.
We are on a new server now and we are not completely done yet, but things are looking better
Here is a list of things that I know are not functioning properly yet:
-
Posting a PM fails
- Posting a PM to multiple people shows an empty screen, but the PMs are posted. Yet there is no record of that in your sent items.
- The background of the edit box (you use to type a message in) is colored
- Some of the drop down menus have a transparent background
- Being the first to comment on a news post gives a blank screen, but the post is made
- Filtering news on the front page is not possible
- The CRPG analyzer does not work (for those of you who knew we had one).
- Surfing via HTTPS still lists the site as insecure (The keys are ordered, hoping to get them soon).
What awful life is without dearest Watch.
