Of Release Cycles and Testing Practices

Discussion in 'General Discussion' started by kaeshiva, Feb 23, 2020.

Thread Status:
Not open for further replies.
  1. kaeshiva

    kaeshiva Avatar

    Messages:
    3,054
    Likes Received:
    11,752
    Trophy Points:
    153
    Gender:
    Female
    This is a summary of items discussed among a group of players, from whom I have been elected the "person to make the forum post."

    The objective of this is NOT to be an attack or criticism, but rather to shed some light on how certain things are perceived within the community, and propose possible changes to the methodology to alter that perception.


    --------------------------------------------------------------------------
    PROBLEM(S)
    1. Broken/bugged content regularly pushed to live
    2. Underutilization of QA/Test environment
    3. Player feedback for related items rarely acknowledged
    --------------------------------------------------------------------------

    Problem 1:

    Players have come to expect low quality: when ‘new content’ is added, players expect it to be non-functional, they expect new loot to not be dropping, they expect new systems to not be working, and the prevailing behaviour is “don’t bother for a couple of weeks until they sort it.”

    “Release cycles” have become meaningless with changes pushed on a random weekday afternoon, perceived lack of structure, i.e. “devs are ‘winging it’” has resulted in lower player confidence.


    Problem 2:

    This problem is three-fold:

    One, the historical “incentive” program was effectively counter-to-purpose, as many players who would have enthusiastically tested for no reward at all stopped doing so when ‘other people’ would receive rare rewards due to ‘random luck.’ This aspect is being addressed with a more suitable system.

    Two, without a scheduled testing phase and with items being pushed to the live build without going through a testing phase, the test environment cannot do its job. Quite often changes never make it on to QA at all, or it isn’t clear when they are added, and the willing cadre of mobilized, ready-to-test players never get the opportunity to test anything. Clear, consistent scheduling of test periods between release cycles is needed, clear patch notes and communication of what has been changed and what the test cases are is needed. It looks like this is starting to happen...this is good!

    Three, and perhaps most significantly, is reported bugs go uncorrected, or worse, completely unacknowledged, and a week later the same bug is reported on live by many more voices resulting most often in the need for a hotfix and several patches which are disruptive to gameplay.


    Problem 3:

    For the various reasons outlined above (poor incentivization, lack of documentation, test cases, lack of acknolwedgement/response of reported bugs/issues), the result is apathy among players who historically provided free testing labor with enthusiasm. While bugs and non-functional content are of course the priority, there are inconsistencies, illogical things, completely counter-intuitive things, (most evident in some of the downright ridiculous crafting recipes) that have been reported multiple times over the years and instead of being fixed, we see more of the same sorts of things added!

    The player perception is: Devs don’t even understand what is in their own game, time and resource is wasted adding completely uneconomical / pointless items that add little to no value to the game and nobody will use them. This is communicated over and over and falls on deaf ears.


    EXAMPLES

    Some of the more infuriating examples:

    • Envy Angel not actually dropping Envy bow until a week after release last year, causing many players to waste time grinding for something that wasn’t actually there.
    • Creepy Doll not actually dropping after being in patch notes and actually dropping briefly, then removed with no communication, causing many players to waste time grinding for something that wasn’t actually there.
    • Early destroyer bear addition being added to a scene that players couldn’t actually get to, causing days and days of players wasting their time on QA and live trying to get something that wasn’t actually there. Players communicated throughout this period that the bears were not in game, and were constantly assured by the dev team that there were. In the end it was revealed they were added to an area to which there was no player access, a rather embarrassing example.
    • Dungeon rooms dropping in the quest version of Hilt instead of the actual version of Hilt, causing players to spend days wasting their time grinding something that wasn’t actually there.

    (There’s a pattern here)

    There have been plentiful other examples of items meant to be in game but not, announced changes actually not being added or not functioning:

    • Warlock chain combines becoming wayland loops
    • The Ardoris Town teleporter, which as its name suggest has one purpose – teleporting the player to Ardoris, was actually teleporting players somewhere else entirely. And this was a $50 purchaseable item!
    • Dozens of new clothing items conflicting/clipping/glitching through bodies and hair in ways so obvious (such as making someone’s entire abdomen DISAPPEAR and you could see through the player) that even cursory testing would have revealed the issues.
    • Skills not working as intended
    • Tooltips being empty, misleading, or flat out wrong

    The general sentiment within the community has shifted from “excited for the patch” (early days) to “don’t bother playing until a week after the patch because everything will be broken, missing, not dropping, or a waste of your time.”

    The purpose of this is not to point the finger at all the things that have been screwed up but to emphasize that in almost all of the above cases, even cursory, lightweight testing would have revealed the problems in a QA environment.


    GENERAL RECOMMENDATIONS

    1. Consider a longer release cycle, or, at least adhere to the established monthly release.

    Any benefit of releasing content ‘early’ is eclipsed by the disillusionment accompanied by how much of it doesn’t work. The overwhelming sentiment is we’d rather have a meaningful monthly release, or even quarterly release, of finished, functional systems, than several mini-patches throughout the month that often break more than they fix.

    2. Establish a consistent schedule of build, test, release and stick to it.

    This means that once things are put in the test environment, sufficient time is allowed to actually test things before they go anywhere near the live environment.

    3. Communicate patch notes, and list test cases

    Utilize your test resource effectively and efficiently. Document changed item(s) and give clear instructions on what you want the player to do – ie, craft this item, kill this monster, visit this scene, try to walk through this rock, etc.

    4. Fix identified bugs before going live, especially the ones that are high visibility/impact

    It is extremely frustrating to the playerbase to see things that they reported on QA go into live without being addressed. The result is they will no longer waste their time reporting. You’re losing your free labor! If it is a low priority bug, at the least acknowledge it and work it into a schedule. It can be difficult to estimate how long a test period will be required to ensure that everything can be fixed, which is why a longer content release cycle may be of benefit.

    5. Listen, analyse, and correct reported issues – improve outdated/problem systems that consistently get reported over and over.

    Several systems within the game need a rethink.

    • Gathering has had no attention at all since pre-persistence, survey is completely pointless, meticulous collection is still questionable. No new tools, skills, etc.
    • Refining skills were meant to be an ‘initial’ pass and over a year no followup – speed increase at grandmaster is barely noticeable (using a stopwatch).
    • The same problems with many crafting recipes exist since pre-persistence
    • Nearly 95% of all cooking recipes serve no purpose/nobody bothers with
    • Virtue system / oracle feedback still vague and largely irrelevant to gameplay
    • In-game book writing/import has been bugged for over a year
    • Vendor UI system needs an overhaul
    • Quest journal needs an overhaul
    • Recipe book just needs to be lit on fire and rebuilt from ground up
    • NPC schedules still annoying/frustrating, players actively avoid towns where these are implemented or grit their teeth when they can’t (due to quest, etc.)
    • Maps are still either missing, partially functional, or at best, laggy.
    • This list could go on and on…
    The point with the above, is that these are all things that could be long term fixes that would benefit from feedback and testing along the way, so that dev time/resource isn’t spent on either solving a problem that doesn’t exist, or implementing a solution that doesn’t address the problem.


    Regarding Revenue Impact
    We understand that the game is a business, and ‘new things for sale’ are necessary to drive revenue. However, there have been negative perceptions when the “monthly content release” consists almost exclusively of ‘new things to buy’ and thousands of issues/problems remain unaddressed.

    Consider the following example case - this is a screenshot of early patch notes a few releases ago where items were highlighted to illustrate the point: "What does each monthly release actually consist of?"

    [​IMG]

    Yellow highlight = new stuff you can buy! Or limited availability.

    Dark yellow highlight = replacement pledge stuff not applicable to a lot of folks.

    Of the 25% of the release notes that were not one of these items, it included the addition of a new recipe that, in its initial iteration, was so uneconomical as to be pointless (subsequently changed), the introduction of the deco palette (which still doesn’t work properly) and the introduction of silver/gold scraps (which are so rare that they serve no purpose). The addition of a skill that, again when added, had no reagent costs and has been subsequently adjusted many times based on player feedback. Feedback that could have/would have (should have?) been delivered via a utilised test environment. Lastly there's mention of agricultural price changes, again a partial implementation affecting only some items with little narrative of what this was meant to achieve or how the new values were derived.

    Again, the purpose is not to point fingers or just wave our arms around angrily, but to give you some insight from how things look on the player side of the fence:

    Yes, we’ve had regularly monthly releases for 74 releases. Awesome.
    • But, in every release half the stuff doesn’t work
    • But, in every release a large % of the new stuff is just stuff to buy
    • A lot of the new stuff that is added is irrelevant to large portions of the playerbase, with significant focus on the “1%” of high levels and combat being the prevailing changes. Sota has so much more to offer than kill-level-loot-repeat!
    • Systems long in need of attention don't even seem to be on the radar

    -----------------------------------------------------------------------------------
    What we'd really like to see:

    Four content releases per year, aligned quarterly, including: (examples)
    • 1 new major system (or old system revitalized) that has been tested and is functional (i.e. “dungeons” – still far from being a complete and functional system!)
    • New recipes, new emotes, new things obtainable in-game (deco, clothes, etc.)
    • 1 new scene or scene revamp
    • Seasonal specials (i.e., the giftboxes and stockings at Christmas that you can craft)
    On the in-between months, and alongside the content releases, monthly releases should include:
    • New things to buy (revenue generation)
    • Small changes/fixes/tweaks/nerfs (yes, we need nerfs sometimes)
    Bug fixing in flight when necessary/critical.



    Reasoning:


    The large content releases would benefit from the extra testing and feedback time. It is much better to release it in a more polished state than to cobble it in week after week. (Do this on QA!!) This just frustrates people and they decide not to bother with it “until later” and a lot of systems seem to get abandoned. Later never happens.

    The in-between releases ensure that new revenue generating items continue to be pushed on a monthly basis.

    Final Thoughts:

    From a player’s viewpoint, it can be difficult to understand the priorities and most end users don’t understand the complexities involved in different desirable changes/features. To them, it just looks like “its all broken, and nobody cares, and its been years and its still not fixed.”

    While we all enjoy shiny new content, its also important to address outdated/legacy systems and bring them up to modern standards, and improvement to core systems can have a much better net positive effect than trying to rush in a half-finished, half-functional “new system” that only a handful of players would even use.

    The recent progress on the bard skill tree is a good example of the right direction - its not ready, so don't release it! This is good. We need more of this. As excited as some of us are about a new skill tree to play with, we'd much rather a functional, tested, balanced (insomuch as is possible) addition than something wacky that is either a) too weak and pointless or b) way overpowered and gets nerfed in a week.


    SotA has something precious – a dedicated community willing to invest their time – free of charge – to provide information, test systems, and offer insight into the player experience for which spreadsheets and metrics are no substitute. Use this resource. But more importantly, use it wisely.

    “Rewards” are a lot less important than just showing appreciation and acknowledgement. The players who live, work, and play in the world you have created understand it very thoroughly. And while there’s always a few selfish bad egg sorts, most of us honestly, earnestly want that world that we live in to be a better place and to be around for years to come.
     
    Last edited: Feb 23, 2020
  2. Vladamir Begemot

    Vladamir Begemot Avatar

    Messages:
    6,194
    Likes Received:
    12,076
    Trophy Points:
    153
    Gender:
    Male
    OMG thank you! You have much diplomacy.

    I would like to add "Prototype!" Slapping a system together in a week is NOT a good method for creating quality content. The first thing you do is prototype several systems and figure out what works. Throw away what does not work. Test the fun system among the players and see if it is fun to them! I fear that Bards are going to be another facepalm because they sprang fully formed from Chris's head.

    Subscribe to Masterclass.com and listen to Will Wright explain the importance of prototypes if needed.

    Longer release cycles would allow time for prototyping.

    Longer release cycles will also allow systems to be finished. I had another "finish something!" post brewing, I'll just let it be put aside thanks to your post. These "a new broken or non-interactive system every month!" methodologies are not working. Finish what you start. Looking at you Lua, dungeons, NPC conversations next month (throwing that one in early, but I'm pretty sure INKY work is basically done now, on to something else) etc.

    And start tying it together. POT's + Sieges + Crafted Gear to Guards = dynamic(!) fun. It really ties the room together. Agriculture + Weather + Fish (as fertilizer, what happened to that one?) = fun. Beds + XP = fun. Player made NPC conversations + emotes = fun .

    Prototype, throw stuff out, test with players, throw more out, find the fun, finish systems, tie them together, now you're rich.

    Thank you again Shimizu, very well said, and what I think so many of us are feeling.
     
    Last edited: Feb 23, 2020
    majoria70, Nefrinia, Mal Hari and 6 others like this.
  3. Elnoth

    Elnoth Avatar

    Messages:
    724
    Likes Received:
    1,557
    Trophy Points:
    93
    If Chris, Richard and Starr invested an hour of their time watching this video (completely different game) they might learn a thing or two about growing a game playerbase through proper game release cadence.

    As it stands right now, I see the game release cadence as being like an annoying leaky tap. Drip, drip, drip, drip.
     
  4. monkeysmack

    monkeysmack Avatar

    Messages:
    632
    Likes Received:
    1,463
    Trophy Points:
    105
    Gender:
    Male
    Location:
    your mom's house
    I agree with the OP wholeheartedly. A very clear and on-point summary of the problem. My gut response to any releases or patches isnt one of excitement but of dread. I think, "Is some part of the game im enjoying now broken?" Fun cancelled. This actually happened last year. I started playing Civ5 instead and didnt come back for 6 months.

    - Monthy release schedule: It feels like this has become a religion. Just let it go and be free. We don't need monthly releases. I don't know of anyone that plays this game just because of the monthly releases. In reality there are actual counter examples as the OP pointed out of people AVOIDING the game because of monthly releases. I think it's fine to release a few store items to purchase on a monthly schedule but messing with major game mechanics is a no no. Especially when they arent properly tested on QA before release.

    - Long standing bugs, maps and quests: Maps and quests are two things new players look closely at when starting an MMO. If you want to improve the new player experience, I would fix these ASAP.
     
  5. Elrond

    Elrond Avatar

    Messages:
    1,277
    Likes Received:
    4,028
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Crafters Town
    Just finished watching it ..so many great insights that would help the team . Bottom line... big expansions bring in big numbers. Advertising properly is also key , but you gotta have content to present to the press . The game presented in the video had expasions named Prophecy or Incursion ours is called what ? Ep 2 ?

    Anyway i think the stuff presented in the video is beyond what our current team can do.

    Pretty much all systems in the game are incomplete .... If i could advise em on something would be this ... Each month release improvements for the systems we currently have... every 3-5 months make a big release ...new systems ...

    October 9 2019 is when Chris took over the project ..thats almost 5 months in which we got

    R 71
    - Decoration Pallete - i used it once or twice in 5 month
    - Wyvern Trophies - spend 2 hours killing some wyverns in 5 months ( no heads no haunches )not for me so moved on
    - Gold and silver scraps - 1 gold scrap from salvaging ? NO TY moving on
    - New Dungeon added Tartarus - fine addition for fighters
    - 1 new enchant and 1 new masterwork
    - Agriculture production was doubled- great for farmers

    R72

    -Lua Integration - I dont know who thats for but i dont think more then 10 players coding here..definetly not for me
    - Unicorn and wyvern food - been 4 months and have yet to be able to buy /make any of these..theyre so rare so again nothing for me here ..moving on to R73 not much else to say of R72

    R 73

    - They added a craftable garden gnome ..rest was just polish on some scenes ...

    R 74

    Things moved a bit this release

    We got a second ring slot ,some new tamable creatures ,some new artifacts and more polish and balance .

    That pretty much sums up whats been worked on for the last 5 months ...i cant find a single feature for my playstyle as a crafter or gatherer that got me excited ... oh well.

    As for the loot ''improvements'' this is what i got today and its been the same for years ... every discussion , every stream, every release same promises '' we will fix the loot...we will improve the loot '' ..and nothing changes .

    [​IMG]
     
    Last edited: Feb 23, 2020
    Osiris3, majoria70, amarious and 13 others like this.
  6. Anpu

    Anpu Avatar

    Messages:
    7,944
    Likes Received:
    9,015
    Trophy Points:
    153
    Location:
    Hemut
    OMG thank you for posting this!

    Please just stop! Stop right now and work only on actual functioning releases. You are a very small development team. We get that. Expecting a release every month just isn’t working. Please just do releases every other month or 4 a year. We are fine with that.

    A release should be tested for at least a month before even being considered for release on live. You could still add content to live once you know it is working correctly on qa. Hell, just ask us.

    A release every month just has to stop. It’s not working and is not what is best for the game currently.

    Properly functioning and fully tested releases (and content) to live needs to be the new direction. Not once a month.
     
    Last edited: Feb 23, 2020
  7. Numa

    Numa Avatar

    Messages:
    2,891
    Likes Received:
    5,620
    Trophy Points:
    153
    Gender:
    Male
    Location:
    Breaker's Landing
    Wholeheartedly agree. A release every quarter would be just fine. Monthly releases neither save time or money - just the opposite. It also doesn't foster goodwill so why keep doing it?
     
  8. Justyn

    Justyn Bug Hunter Bug Moderator

    Messages:
    1,216
    Likes Received:
    701
    Trophy Points:
    113
    Thank you for taking the time to put together what we have all been thinking and saying for a long time. I +1 all of this! I may log in on the weekend after a release, but it is always with the thought that I'm going to have to get in, do what needs to be done and get out because there will be no point to doing anything meaningful until after the Monday patch.
     
  9. Nexus-6

    Nexus-6 Avatar

    Messages:
    181
    Likes Received:
    473
    Trophy Points:
    30
    Gender:
    Male
    Location:
    Wild East
    Meanwhile after cycle change...
    3 months passed and not a single release? WTF?!? What developers do ... :p
     
  10. KnightFalz

    KnightFalz Avatar

    Messages:
    95
    Likes Received:
    112
    Trophy Points:
    8
    Good post.
     
  11. Crazy Teeka

    Crazy Teeka Avatar

    Messages:
    50
    Likes Received:
    70
    Trophy Points:
    8
    The DEVS need to look at ALL chairs and beds, and fix the ones that have the avatar in the wrong position, some chairs have this issue, some beds have the avatar positioned off the side of the bed. This might sound minor but it's been put off for years. Everyone has just accepted thats how they work. No one cares about the chairs and the beds.

    Even "Fallout 76" got the chairs and the beds right and thats a game with a reputation of being full of bugs.
     
    Last edited: Feb 24, 2020
  12. Senash Kasigal

    Senash Kasigal Avatar

    Messages:
    1,202
    Likes Received:
    2,685
    Trophy Points:
    125
    Gender:
    Male
    Location:
    Germany
    Great Post. thx

    That was the Beginning of the End (for me).
    First, I refused to do Guild Events at weekends after a Release, than I played other Games at this weekends, had fun in these Games and stopped completely playing SotA. Now I am only following the News and hope for some good changes.
     
  13. Gorthyn

    Gorthyn Avatar

    Messages:
    617
    Likes Received:
    1,497
    Trophy Points:
    93
    An excellent post by Kaeshiva as always.
     
    Rinaldi, Mal Hari and kaeshiva like this.
  14. Nexus-6

    Nexus-6 Avatar

    Messages:
    181
    Likes Received:
    473
    Trophy Points:
    30
    Gender:
    Male
    Location:
    Wild East
    So...

    1. We (not all but most) already paid for e2-ep5... ep2 we taking in 2020... When ep3? ep 5? Never because devs "move chairs and beds" ?? But we PAID for them...

    2. I play many MMO and no one of then be "perfect", "finish", "polish" etc... MMO, for me, have 2 state - playable or unplayable... SotA is playable, as for me...

    3. Changing cycle has zero effect... Release every 3 month instead 1 doesn't mean developers will work better or we have more content...

    BUT...

    I agree - after ep2 2-3 month (releases :p) devs must spend for quality of life work...
     
    Mal Hari likes this.
  15. Griffler

    Griffler Avatar

    Messages:
    146
    Likes Received:
    220
    Trophy Points:
    30
    Beautifully crafted, thank you for the time you took to put all this together @kaeshiva , Couldnt have put it better myself!

    @Chris @DarkStarr Everyone on the team should read this and meditate on it!
     
    Elnoth, Vero, Minerva and 4 others like this.
  16. Elgarion

    Elgarion Dev Emeritus Dev Emeritus

    Messages:
    3,223
    Likes Received:
    9,292
    Trophy Points:
    153
    Gender:
    Male
    There is some really good stuff in this thread, thanks :) FYI, @Ravalox is working hard to improve your experiences with the Player Test Environment (formerly known as Princ…. errr… QA I mean). And one of the big things is that it should be up 24/7 all month long each release for larger windows for testing.
     
  17. Curse2

    Curse2 Avatar

    Messages:
    49
    Likes Received:
    88
    Trophy Points:
    8
    You had me on Problem 1

    What an eloquent, passionate, and well thought out communication of facts and effects. Only thing I would have added is a "How To" section on running a business...

    On this note, I would ask for your support in forming a "FORMAL"...

    PLAYER BOARD - call it whatever you like, empower it to make real decisions...

    Not a behind the scenes group of messages. texts. and phone calls, from a hidden and constantly suspect sphere of influence; a formal board that actually helps the company succeed.

    @kaeshiva Would you be willing to serve on such a board?

    Business people are needed, old players are needed, new players are needed, Crafters, PVPers, Decorationists, Marketing, Advertising, you name it, on a list of helpful skills.

    One of the primary roles should be to pave the way to allow player at large voting on the direction of some of the work content (AKA prioritizing what's next from a player perspective).

    Could go on and on...

    Sorry for my rabbit hole post in response to your most excellent summation of the past and current state of things!
     
  18. Mal Hari

    Mal Hari Avatar

    Messages:
    266
    Likes Received:
    742
    Trophy Points:
    28
    Gender:
    Male
    Location:
    The Jade Valley
    Wow and thank you for saying what needed to be said and sadly has been said so many times.

    Wish there was more for me to say about it all. But I'm one of the long term folks that's getting burned out with disappointment.
     
  19. GMDavros

    GMDavros Avatar

    Messages:
    290
    Likes Received:
    391
    Trophy Points:
    28
    Bravo @kaeshiva. As an IT professional in software development and delivery, I concur with your points 100%. Well meaning, and well said.

    Fenris
     
  20. Gwendolyn Obscuro

    Gwendolyn Obscuro Bug Hunter

    Messages:
    594
    Likes Received:
    1,142
    Trophy Points:
    63
    Thank you for this post. We all know the devs work hard and we can be impatient for new content but some changes need to be made to give a better impression to new players.

    I'm glad QA is up 24/7 now but that seems to be used as an excuse to push any content that has been on QA for a day to live immediately. QA is great for testing these new systems but the new content needs to stay on QA and not move to live unless it is actually ready for live, no matter if it is release week or players keep asking for it. You can only do so much with a small team but it is better to do less well than to do more in such a buggy way that we constantly have to tell new players "yeah that's broken, maybe they'll fix it some day"

    A couple of recent frustrating examples:

    2nd Ring Slot
    • We were all excited to get a second ring slot
    • Atos said we would not be able to use two of the same artifact- OK, makes sense!
    • Ring slot was pushed to live knowing we could still use two ring slots (should have been restricted first!)
    • A patch was released on QA to eliminate double artifacts and bugs were reported that it didn't work on many items
    • The patch notes on live announced that this has been fixed and only one of an artifact can be used
    • Currently on live you can still use two of the same ring/wand except for a few seemingly random cases and players are used to the extra power now. So when this is changed it will be viewed as a nerf.
    NPC Conversationalists
    • These conversationalists will be great for allowing user created content. While a lot of players thought this meant adding dialog to existing NPCs we are still excited to see how this turns out.
    • The conversation feature was accidentally added to an NPC that could not be sold in store. oops, but ok.
    • Elgarion found a way around this oops by handing out automatons on QA for testing and even had a stream where several of us tested and found several significant bugs on QA.
    • The system was released to live even with all the problems above.
    • Additional NPCs were quickly added to the crown shop (directly to live, skipping QA) and this feature was announced as released.
    • Currently on live the NPC conversationalists do not act the same for someone visiting your lot when you are not there compared to how they act when you are standing there. I know this is just the first phase of Inky but that should mean that new features/options will be released later for them not that the features already released don't work yet.
     
Thread Status:
Not open for further replies.