Of Release Cycles and Testing Practices

Anpu · Feb 25, 2020

TaiKee Vetrae said: ↑

I would love to help in QA and test things out, I just currently do not see the point because a) things are only there for 1-2 days before they go live;

Click to expand...

You might not have noticed this yet @TaiKee Vetrae but the new QA forums, now called Player Test Environment (QA Server) are now up. And this seems now to be a permanent forum (it's no longer moved when a release happens).

Currently you can make bug reports and feedback and these posts will be housed there. From what I can guess, this will now serve as a basic repository for game issues (bugs). And Ravalox has been doing a great job in adding issues there to assigned JIRA numbers. So at least hopefully, some of these issue will get looked at or possibly fixed. So you can right now, head off there and start reporting things, if you want (I have been).

I'm sure that Ravalox is probably going to be making a post on how this new Player Test Environment will be and what the intent and purpose of it is, and what they want and need from us. I'm just waiting for that post, and totally get they are all busy. I'm a patient guy.

I DO know that Ravalox would like people who are having those Decoration issues (this is only happening to some people), to check on QA right now and see if that is fixed. It should be. I sadly do not have this issue so cannot check that.

Numa · Feb 25, 2020

IMO this was the perfect time for such a post, it just happened that it was @kaeshiva who made the post and did it well.

Sometimes timing is everything.

Sulaene Moon · Feb 25, 2020

Same issues, different day. Do we really think that things will change?

that_shawn_guy · Feb 25, 2020

Sulaene Moon said: ↑

Same issues, different day. Do we really think that things will change?
Click to expand...

Change is always a possibility. Sometimes intentional. Sometimes accidental.

Anpu · Feb 26, 2020

We should probably stick to the thread topic, and keep that discussion going.

Gadfrey Zukes · Mar 16, 2020

As one of the quiet, non-guildie, mostly solo, non min-maxer players, I just want to add that I totally agree with @kaeshiva’s post and appreciate the constructive way it was written.

I’d also like to add that imo the team could do away with all the hours of live streams that seem to have 5 mins of interesting content, no gameplay examples, and no focus - and instead use some of those hours to craft/edit a single weekly 20 min max video that stays tight on a topic, or provides updates/news, or “how to” videos on some of the more complex systems. (less about fishtanks, allergies, prize winners, and daycare and more about the game, the designers, and the work.) otherwise, I would suggest just do away with those videos altogether and spend the time in game with players playtesting systems. (I also don’t think the fishing system needs any more livestreamed playtesting)

I enjoy the world that you have built and especially look forward to polished additions of some of the systems like “ink” that may allow for fun, meaningful player content.

Kabalyero Kidd · Oct 31, 2020

kaeshiva said: ↑

This is a summary of items discussed among a group of players, from whom I have been elected the "person to make the forum post."

The objective of this is NOT to be an attack or criticism, but rather to shed some light on how certain things are perceived within the community, and propose possible changes to the methodology to alter that perception.

--------------------------------------------------------------------------
PROBLEM(S)
1. Broken/bugged content regularly pushed to live
2. Underutilization of QA/Test environment
3. Player feedback for related items rarely acknowledged
--------------------------------------------------------------------------

Problem 1:

Players have come to expect low quality: when ‘new content’ is added, players expect it to be non-functional, they expect new loot to not be dropping, they expect new systems to not be working, and the prevailing behaviour is “don’t bother for a couple of weeks until they sort it.”

“Release cycles” have become meaningless with changes pushed on a random weekday afternoon, perceived lack of structure, i.e. “devs are ‘winging it’” has resulted in lower player confidence.

Problem 2:

This problem is three-fold:

One, the historical “incentive” program was effectively counter-to-purpose, as many players who would have enthusiastically tested for no reward at all stopped doing so when ‘other people’ would receive rare rewards due to ‘random luck.’ This aspect is being addressed with a more suitable system.

Two, without a scheduled testing phase and with items being pushed to the live build without going through a testing phase, the test environment cannot do its job. Quite often changes never make it on to QA at all, or it isn’t clear when they are added, and the willing cadre of mobilized, ready-to-test players never get the opportunity to test anything. Clear, consistent scheduling of test periods between release cycles is needed, clear patch notes and communication of what has been changed and what the test cases are is needed. It looks like this is starting to happen...this is good!

Three, and perhaps most significantly, is reported bugs go uncorrected, or worse, completely unacknowledged, and a week later the same bug is reported on live by many more voices resulting most often in the need for a hotfix and several patches which are disruptive to gameplay.

Problem 3:

For the various reasons outlined above (poor incentivization, lack of documentation, test cases, lack of acknolwedgement/response of reported bugs/issues), the result is apathy among players who historically provided free testing labor with enthusiasm. While bugs and non-functional content are of course the priority, there are inconsistencies, illogical things, completely counter-intuitive things, (most evident in some of the downright ridiculous crafting recipes) that have been reported multiple times over the years and instead of being fixed, we see more of the same sorts of things added!

The player perception is: Devs don’t even understand what is in their own game, time and resource is wasted adding completely uneconomical / pointless items that add little to no value to the game and nobody will use them. This is communicated over and over and falls on deaf ears.

EXAMPLES

Some of the more infuriating examples:

Envy Angel not actually dropping Envy bow until a week after release last year, causing many players to waste time grinding for something that wasn’t actually there.

Creepy Doll not actually dropping after being in patch notes and actually dropping briefly, then removed with no communication, causing many players to waste time grinding for something that wasn’t actually there.

Early destroyer bear addition being added to a scene that players couldn’t actually get to, causing days and days of players wasting their time on QA and live trying to get something that wasn’t actually there. Players communicated throughout this period that the bears were not in game, and were constantly assured by the dev team that there were. In the end it was revealed they were added to an area to which there was no player access, a rather embarrassing example.

Dungeon rooms dropping in the quest version of Hilt instead of the actual version of Hilt, causing players to spend days wasting their time grinding something that wasn’t actually there.

(There’s a pattern here)

There have been plentiful other examples of items meant to be in game but not, announced changes actually not being added or not functioning:

Warlock chain combines becoming wayland loops

The Ardoris Town teleporter, which as its name suggest has one purpose – teleporting the player to Ardoris, was actually teleporting players somewhere else entirely. And this was a $50 purchaseable item!

Dozens of new clothing items conflicting/clipping/glitching through bodies and hair in ways so obvious (such as making someone’s entire abdomen DISAPPEAR and you could see through the player) that even cursory testing would have revealed the issues.

Skills not working as intended

Tooltips being empty, misleading, or flat out wrong

The general sentiment within the community has shifted from “excited for the patch” (early days) to “don’t bother playing until a week after the patch because everything will be broken, missing, not dropping, or a waste of your time.”

The purpose of this is not to point the finger at all the things that have been screwed up but to emphasize that in almost all of the above cases, even cursory, lightweight testing would have revealed the problems in a QA environment.

GENERAL RECOMMENDATIONS

1. Consider a longer release cycle, or, at least adhere to the established monthly release.

Any benefit of releasing content ‘early’ is eclipsed by the disillusionment accompanied by how much of it doesn’t work. The overwhelming sentiment is we’d rather have a meaningful monthly release, or even quarterly release, of finished, functional systems, than several mini-patches throughout the month that often break more than they fix.

2. Establish a consistent schedule of build, test, release and stick to it.

This means that once things are put in the test environment, sufficient time is allowed to actually test things before they go anywhere near the live environment.

3. Communicate patch notes, and list test cases

Utilize your test resource effectively and efficiently. Document changed item(s) and give clear instructions on what you want the player to do – ie, craft this item, kill this monster, visit this scene, try to walk through this rock, etc.

4. Fix identified bugs before going live, especially the ones that are high visibility/impact

It is extremely frustrating to the playerbase to see things that they reported on QA go into live without being addressed. The result is they will no longer waste their time reporting. You’re losing your free labor! If it is a low priority bug, at the least acknowledge it and work it into a schedule. It can be difficult to estimate how long a test period will be required to ensure that everything can be fixed, which is why a longer content release cycle may be of benefit.

5. Listen, analyse, and correct reported issues – improve outdated/problem systems that consistently get reported over and over.

Several systems within the game need a rethink.

Gathering has had no attention at all since pre-persistence, survey is completely pointless, meticulous collection is still questionable. No new tools, skills, etc.

Refining skills were meant to be an ‘initial’ pass and over a year no followup – speed increase at grandmaster is barely noticeable (using a stopwatch).

The same problems with many crafting recipes exist since pre-persistence

Nearly 95% of all cooking recipes serve no purpose/nobody bothers with

Virtue system / oracle feedback still vague and largely irrelevant to gameplay

In-game book writing/import has been bugged for over a year

Vendor UI system needs an overhaul

Quest journal needs an overhaul

Recipe book just needs to be lit on fire and rebuilt from ground up

NPC schedules still annoying/frustrating, players actively avoid towns where these are implemented or grit their teeth when they can’t (due to quest, etc.)

Maps are still either missing, partially functional, or at best, laggy.

This list could go on and on…

The point with the above, is that these are all things that could be long term fixes that would benefit from feedback and testing along the way, so that dev time/resource isn’t spent on either solving a problem that doesn’t exist, or implementing a solution that doesn’t address the problem.

Regarding Revenue Impact
We understand that the game is a business, and ‘new things for sale’ are necessary to drive revenue. However, there have been negative perceptions when the “monthly content release” consists almost exclusively of ‘new things to buy’ and thousands of issues/problems remain unaddressed.

Consider the following example case - this is a screenshot of early patch notes a few releases ago where items were highlighted to illustrate the point: "What does each monthly release actually consist of?"

Yellow highlight = new stuff you can buy! Or limited availability.

Dark yellow highlight = replacement pledge stuff not applicable to a lot of folks.

Of the 25% of the release notes that were not one of these items, it included the addition of a new recipe that, in its initial iteration, was so uneconomical as to be pointless (subsequently changed), the introduction of the deco palette (which still doesn’t work properly) and the introduction of silver/gold scraps (which are so rare that they serve no purpose). The addition of a skill that, again when added, had no reagent costs and has been subsequently adjusted many times based on player feedback. Feedback that could have/would have (should have?) been delivered via a utilised test environment. Lastly there's mention of agricultural price changes, again a partial implementation affecting only some items with little narrative of what this was meant to achieve or how the new values were derived.

Again, the purpose is not to point fingers or just wave our arms around angrily, but to give you some insight from how things look on the player side of the fence:

Yes, we’ve had regularly monthly releases for 74 releases. Awesome.

But, in every release half the stuff doesn’t work

But, in every release a large % of the new stuff is just stuff to buy

A lot of the new stuff that is added is irrelevant to large portions of the playerbase, with significant focus on the “1%” of high levels and combat being the prevailing changes. Sota has so much more to offer than kill-level-loot-repeat!

Systems long in need of attention don't even seem to be on the radar

-----------------------------------------------------------------------------------
What we'd really like to see:

Four content releases per year, aligned quarterly, including: (examples)

1 new major system (or old system revitalized) that has been tested and is functional (i.e. “dungeons” – still far from being a complete and functional system!)

New recipes, new emotes, new things obtainable in-game (deco, clothes, etc.)

1 new scene or scene revamp

Seasonal specials (i.e., the giftboxes and stockings at Christmas that you can craft)

On the in-between months, and alongside the content releases, monthly releases should include:

New things to buy (revenue generation)

Small changes/fixes/tweaks/nerfs (yes, we need nerfs sometimes)

Bug fixing in flight when necessary/critical.

Reasoning:

The large content releases would benefit from the extra testing and feedback time. It is much better to release it in a more polished state than to cobble it in week after week. (Do this on QA!!) This just frustrates people and they decide not to bother with it “until later” and a lot of systems seem to get abandoned. Later never happens.

The in-between releases ensure that new revenue generating items continue to be pushed on a monthly basis.

Final Thoughts:

From a player’s viewpoint, it can be difficult to understand the priorities and most end users don’t understand the complexities involved in different desirable changes/features. To them, it just looks like “its all broken, and nobody cares, and its been years and its still not fixed.”

While we all enjoy shiny new content, its also important to address outdated/legacy systems and bring them up to modern standards, and improvement to core systems can have a much better net positive effect than trying to rush in a half-finished, half-functional “new system” that only a handful of players would even use.

The recent progress on the bard skill tree is a good example of the right direction - its not ready, so don't release it! This is good. We need more of this. As excited as some of us are about a new skill tree to play with, we'd much rather a functional, tested, balanced (insomuch as is possible) addition than something wacky that is either a) too weak and pointless or b) way overpowered and gets nerfed in a week.

SotA has something precious – a dedicated community willing to invest their time – free of charge – to provide information, test systems, and offer insight into the player experience for which spreadsheets and metrics are no substitute. Use this resource. But more importantly, use it wisely.

“Rewards” are a lot less important than just showing appreciation and acknowledgement. The players who live, work, and play in the world you have created understand it very thoroughly. And while there’s always a few selfish bad egg sorts, most of us honestly, earnestly want that world that we live in to be a better place and to be around for years to come.
Click to expand...

WOW! This is like a guide to a successful MMO or Live Service Game...

the Lacedaemonian · Nov 13, 2020

Kabalyero Kidd said: ↑

WOW! This is like a guide to a successful MMO or Live Service Game...
Click to expand...

And here we are.

Log in

Of Release Cycles and Testing Practices

Anpu Avatar

Numa Avatar

Sulaene Moon Avatar

that_shawn_guy Bug Hunter

Anpu Avatar

Gadfrey Zukes Avatar

Kabalyero Kidd Avatar

the Lacedaemonian Avatar

Log in

Of Release Cycles and Testing Practices

Anpu Avatar

Numa Avatar

Sulaene Moon Avatar

that_shawn_guy Bug Hunter

Anpu Avatar

Gadfrey Zukes Avatar

Kabalyero Kidd Avatar

the Lacedaemonian Avatar

Useful Searches