R106 Restoration and Recovery efforts

What Happened?

When we deployed R106 on the morning of September 29th, we found that there was underlying database corruption. We worked diligently for the next week identifying the issues and attempting to recover the database. During this time we worked on a number of past iterations restored from our backups.

After seven days of work we still unable to achieve stability and made the very difficult choice to restore a database backup saved prior to releasing R105.

The type of restore is known as a 'Rollback'. This means that any and all progress gained from the point of the restored backup will have been lost. This also means that new characters created will no longer exist.


What's my impact?

A rollback is like using a time machine. When the Live game server was restored and brought on-line, all progress, stats, etc. are as they were on August 24th at 8am CT. This impacts the following in addition to other aspects of your character's status:

New Characters: If you created a new character during this window it will be missing, please re-create the character and reach out to support for assistance in having any earned XP reallocated to your XP Pool.

XP and Gold: We were able to extract information from the most current backup, which allowed us to restore player account XP levels. We are investigating what we can do to restore player account gold.

Skills: XP has been placed on your account in the XP pool. We are unable to restore XP to specific skills, so any skill level changes to your character will need to be re-trained.

COTOS/Website Purchases: Any COTOs or store items purchased from 8/24 - 9/29 and throughout the outage will be (re) delivered via the Rewards Chest in game.

Houses/Lots: If you claimed a lot or placed a house during the period of time the rollback covers, you will need to reclaim.

POTs: IF you upgraded or purchased a New POT during the time frame, we are working to re-apply the new or upgraded town to the game.

NOTE: We cannot restore decoration work done on lots or in POTs.

Taxes/House expiration timer:
We have suspended the eviction timer and will keep that in place until at least November 18th. During this time we will be working to determine the best feasible way to address back taxes.

If the deed was obtained prior to 8/24 @ 0800 CT then the deed should be where it was previously: in your bank, Inventory, or storage.


Rewards?

Please note that Subscriber and Login rewards have been re-delivered as part of the R106 bundles. Due to the game's structure we cannot re-deliver R105 and R106 rewards separately, so the Releases have been combined into one bundle. Some additional items have been added to the October Login Rewards.


What's next?

We plan to have a Livestream to go into more detail surrounding the issues and the recovery efforts as soon as possible. We will tweet out details as soon as confirmed.

As noted above there are still aspects we are working on to make accounts as whole as possible.
We will discuss all of this in detail on the livestream, along with other aspects that have impacted the players.

We will post any updates here on this thread. Thank you for being so patient during this calamity!

 

Update:

We continue to work on post restore cleanup (there is a Live patch on the way now, and most likely others to come as we sort things out).

There will be a stream this Friday (Oct 14 @ 4pm CT). We will provide info on what happened, what we did to recover, and what has yet to be done. We will be doing a Q & A session during this stream as well.

There is a flurry of activity as we work to get things normalized. Thank you once again for being patient while we work through this.

 

Update2:

We are putting together the process for the following items:
* Recovering gold on affected accounts
* Clearing the back-taxes on affected accounts

Please note that the expiration timer has been "suspended", so houses should not fall while we work to clear the back-taxes on the lots. Purging the back taxes will require taking the Live server offline. We will discuss the details of this during the Friday Livestream.

Did you buy items with COTOS in the Crown store during the timeframe we rolled back over? Though the COTOS will still be in your account, and any COTOS purchased in September will also automatically be delivered to you via the Rewards Chest. We understand that some of the items were purchased when they were on sale, or were available during a Vault or seasonal cycle making it near impossible to re-purchase them. We will be announcing our plans on how we will assist in allowing players to re-obtain these items during the Friday stream!

 

Addendum:

• NOTE: We are suspending the eviction timer until Nov 18th, not suspending taxes. The daily increment for lot taxes will continue as normal. The extension of the timer allows us to have time to clear the back taxes on lots without having lots fall. After your back taxes are cleared, you should start populating tax credit funds as normal. If you have paid off your back taxes instead of waiting for the scripted back tax purge, please reach out to support to have those funds refunded (we'll verify the amount in the server logs).

 

LiveStream summary:

The Livestream can be seen here on YouTube:


Highlights
(starting at 9:00):
* We have had virtually no down time in over five years
* This outage lasted a little over a week
* We could have restored the old data (from 8/24) right away, or we could have forged forward with the data post Release 106, which was in very bad shape.
It was only after 5 days of digging deeper and deeper into the issues (correcting as much as we could as we went) did we realize that the issues spidered out so much that recovery was not feasible.
* We were not hacked.
* We were not blackmailed for Bitcoin
(11:30)
* The root cause is a problem that has been around for years, we just had never run into this situation. In this case the only errors seen appeared to be minor and were not impacting server stability or game play. The servers remained up and stable until we rebooted them as part of the normal process for deploying a new Release. This process flushes the cache and is when the corruption revealed itself.
We do back up the data (multiple times a day), but the data that was being backed up was already corrupted.
* The initial visible issue was that player lots appeared to be unclaimed, but in fact were still associated in the DB. we were able to correct that problem, but it revealed another underlying issue, then another, etc.
* Ended up restoring to a point where the DB was stable.

Recovery/Compensation:
Experience Earned has been restored
Looking to compensate for Gold earned. (still working on implementing this)
Crowns and any web store items bought during that time will auto re-deliver on login
Back Taxes are being cleared on Monday 10/17 at 0700 CT (this requires brief server downtime!)
We have pretty much everything on sale (through the end of R106) including Deed Upgrades in game.

We understand the impact of the time spent that has been un-done. The pain of the hours/days spent with Deco, Skill building, farming, and for newer players they even lost their characters. We have made every effort to restore, recover or compensate for as many aspects as we feasibly can.

Mitigation:
We are changing processes, and implementing new hardware structure to ensure this never occurs again. The servers will all be migrated to Virtual Machines (soma already are on VM, but we do have dedicated servers that need to be migrated to increase resiliency.)
This plus the implementation of Unity 2021 and the new Build machines will provide a mirrored infrastructure allowing us to swap over in the event anything as catastrophic as this occurs.

Chris will be coming back to SotA almost full time in November. We have a ton of infrastructure work to do in addition to improving the game itself. These plans can easily take up the entirety of November to put in place.

Community:
Chris and the entire team wants to thank the community for their wonderful support during this time.


Please watch the video for full details!

 

Updates (Oct 19):

We are heading into the home stretch for recovery tasks. We still have some bonus things that need to be created to be added to the login reward that may or may not make it into the R107 release.

* The Gold recovery data has been compiled and a testing directive should be sent out by @DavidDC shortly to test/check it on QA before we apply the adjustments to the live server.

* REMINDER!! We have applied the past due tax fixes to the live server (applied on Monday 10/17). So please pay attention and pay lot taxes if they are due or apply credit funds as normal.
We currently have the expired house grace timer extended. but this will be reset to it's default setting as of November 18th, and with that, this phase of recovery will be complete.

* We are holding a Live Dev event in Hometown on October 22nd. This is to assist players with missing items purchased during the rollback period that cannot be re-purchased again because they are no longer visible in the crown store or are no longer on sale
Please see the thread on this topic and answer the poll to assist us to staff acordingly: https://www.shroudoftheavatar.com/f...nd-chance-live-sale-hometown-on-10-22.172772/

 

Gold Recovery is now on the QA server: https://www.shroudoftheavatar.com/forum/index.php?threads/testing-directive-gold-restoration.172821/
If able, please log in on the server and check for system sanity, thanks!