Problems with QA Testing

Discussion in 'General Discussion' started by Rinaldi, Jan 27, 2020.

  1. Rinaldi

    Rinaldi Avatar

    Messages:
    851
    Likes Received:
    2,686
    Trophy Points:
    93
    Gender:
    Male
    Location:
    Jade Valley & Wizards Rest
    The intent of this post is to provide constructive feedback and to help SOTA improve and be successful.

    I am a long time bug reporter. For 4 years, I have tested on almost every release of the QA and public server. Recently, I have greatly curtailed my bug testing for the reasons listed below. It is difficult for me to justify using my valuable time on QA as long as these issues persist.
    I believe QA testing is and will continue to be insufficient because of the problems listed below.

    1. No patch release notes - this is a critical issue. How are players supposed to know what to test if they don't know which features/items have been added? Note: commenting about QA during a live-stream is not a satisfactory replacement for formal patch notes. QA Patch notes should be released promptly with each patch. If a feature is only partially implemented, then the extent of its implementation should be stated so that players do not waste time testing incomplete features, and devs don't have to sort through needless bug reports on the incomplete features.
    2. Unaddressed bug reports (no response from devs) - this was a huge issue before Ravalox joined the team. Massive amounts of bug reports went completely unaddressed, despite being reported multiple times over many releases. And while there has recently been a tremendous improvement because of Ravalox, there are still bug reports that are unaddressed.
    3. Poor communication regarding the bug testing rewards program - is this program still in effect? I asked this question on a recent live-stream and the response, which was both uninformative and disappointing, was basically 'I don't know, maybe someone was involved with that at some time'. I have asked several people who frequently test on QA and none of them have received rewards for over 1 year (I know rewards are done by lottery, but my informal poll suggests a defunct reward system). A clear and definitive update statement should be posted in the forum and announced on live-stream regarding whether the bug reward program is still in effect. If rewards are being given out, then create a simple monthly post stating "bug rewards were issued for this month." (and if time permits, list the reward & the player who received it).
    4. Not much Incentive to test on QA - Many people test on QA simply because they want to improve the game and they don't expect anything in return (myself included: I tested before and after bug rewards were a thing). However, I do think that some compensation for testing would be nice. In effect, the time a player spends on QA testing is time that could have been spent on advancing his/her avatar in the public release. So, QA testing may occur at the expense of one's avatar. The bug rewards program did help provide some compensation, but only a few testers are given rewards each release. Perhaps consider something like this: every person that tests on QA for 4 hours gets 1 million adventure XP and 1 million producer XP on their public server avatar (at least this partly offsets the lack of character advancement while QA testing). In addition, you could continue to issue the bug rewards by the current lottery system.

    @Chris, @Elgarion, @Ravalox
     
    Last edited: Jan 27, 2020
  2. Sentinel2

    Sentinel2 Avatar

    Messages:
    989
    Likes Received:
    1,631
    Trophy Points:
    93
    Agreed.

    It's been many months since I submitted a bug report. There are a number which have gone unanswered. I have no idea if they are being worked on. And yes I've rechecked and found the same bugs in place release after release. I have more but not sure I'm benefiting anyone by posting.

    Regarding release notes. I have no idea what to test, cept what I hear from the twitch streams. And that's when I have time to even watch them. Now we have twitter feeds. I check once in a while and try to filter for bugs and new features they talk about. But now I'm chasing THEM down to find out what THEY did so I can help.

    Then the last point. I do it for the hell of it because I care. There used to be incentives. Haven't seen them in years. So I do what I can and may be working ineffectively. With my limited time as it is. I think people like me can be better utilized. We're willing. We just need to be pointed in the right direction.
     
  3. Anpu

    Anpu Avatar

    Messages:
    3,724
    Likes Received:
    4,161
    Trophy Points:
    153
    I personally don’t care about the rewards. Just remove it.

    The people that actually care, will continue to test and report bugs even if there’s no rewards (like myself).

    The rest of the op’s statements I do agree with. Some clear and better communication for the QA process would be most welcome. I’m certain we will get to that point someday. It will just take a bit of time.
     
    Lesni, CatherineRose, Vero and 3 others like this.
  4. Barugon

    Barugon Avatar

    Messages:
    12,186
    Likes Received:
    19,307
    Trophy Points:
    153
    Gender:
    Male
    I would like to see the experience that you earn on QA (both adventurer and producer) put into the experience pools on live.
     
  5. Datendrache

    Datendrache Avatar

    Messages:
    290
    Likes Received:
    1,314
    Trophy Points:
    40
    Gender:
    Male
    Another issue that been driving me nuts over the last year.

    Story:

    I wander into a random scene on the QA server and find a player there. The player tells me they are stuck and explains they were fishing a resource node. I help, verify the issue and convince the player to post on the forums.

    This issue gets bumped to Critical. https://www.shroudoftheavatar.com/f...e-in-east-sanctus-forest.156634/#post-1242858

    And then... nothing. I mean, this player deserves a reward, deserves encouragement to test, and especially deserves a big pat on the back for 1) deciding to come on QA and 2) actually finding a critical issue.

    Do I know if they are disappointed? No. Do I know if they are still playing? No. Do I know if this whole episode ever amounted to anything with that player? No. That was their only post.

    But I can easily recognize a huge failure to not do more, where a spark of interest drove a player to go through the trouble to install QA and play for the very first time and actually accomplish something. As an older player, I can understand its easy to shake a finger at me and say I don't need any more of these rewards, but I've always been trying to support the underdogs of this game, and its them who get their hopes up only to be met with -highly avoidable- and regrettable situations like this one. And I've tried, on multiple occasions, to light this fire under @Chris and even find people who would volunteer to hand out these if Chris allowed it -- so far nothing has worked. But it would be better to take this program down than to let people believe they're going to get something for 4 hours of testing when they aren't.

    Engineer
     
  6. FrostII

    FrostII Avatar

    Messages:
    4,644
    Likes Received:
    9,356
    Trophy Points:
    153
    Gender:
    Male
    Location:
    Pacific Northwest
    @Rinaldi I agree completely with all 4 of your points ! They are SPOT ON !
    Most especially #1, and I'll quote you here :
    This is the primary reason why I stopped wasting my time (with the exception of the recent challenge that @Chris issued for testing changes to UT and was part of the team of high level players that accepted and won that challenge .)

    Since the beginning of QA testing (and I go back over 5 yrs of it now) there have been no "Patch Notes" for QA, and we've always been "left in the dark" regarding what HAS BEEN included and what hasn't, with the end result of testing for hours and hours only to find out that changes had not been made yet, and more typically were not even added until the next Release.

    I (we) are very happy to test, but doing so without being made aware of whether or not the changes have made it in - only to find out that literally hours of my (our) time has been completely wasted, has driven us away in frustration and disbelief that this is not recognized by Port as an issue.

    And, as Rinaldi pointed out, having the only communication about what is being worked on in a brief "live" statement (in a 1 to 2 hour stream) is (respectfully) NOT the same as up to date patch notes that we can quickly access and be on our way testing.
    Add a "QA Patch Notes" thread under the QA Feedback & Bug Reports forum listing !

    We are ALL more than willing and ready to give feedback and bug reports on game changes/additions - but, as Rinaldi also so aptly stated:

    If you're @Chris concerned that giving rewards or xp will be abused by some, then you have the ability to see exactly what they did for their 4hrs and subsequently ban them from any rewards when it's apparent that they are abusing it.

    Hoping for better communication in the future - that ALL have access to... *cough - live stream* - in the way of up to date patch notes, so we can work together to make SotA successful for years to come. The BULK of us are either working or sleeping when you do your streams, and - since (apparently) no one there has heard of voice to text apps that could be posted with little time editing - we are left the choice of using a couple of hrs to listen to a recorded stream for info we're interested in - or playing the game with our limited time.

    Surely, you must see where we're coming from on these points.........

    Respectfully,
    Robert Frost
     
    Last edited: Jan 27, 2020
    Mhtic, Cordelayne, Lesni and 2 others like this.
  7. Sentinel2

    Sentinel2 Avatar

    Messages:
    989
    Likes Received:
    1,631
    Trophy Points:
    93
    Rewards were nice. But I agree. If the devs will fix the bugs that's one thing.

    Submitting bug after bug and not hearing anything. I dunno. Eventually people stop. I can run on the mouse wheel for only so long without feedback.
     
    CatherineRose, Rinaldi, Anpu and 2 others like this.
  8. Ravalox

    Ravalox Dev At Large Moderator SOTA Developer

    Messages:
    609
    Likes Received:
    1,429
    Trophy Points:
    105
    Gender:
    Male
    Location:
    Austin, TX
    Things have been out of whack for some time on QA. This is going to change; plans are in place and things are shifting around to make this happen, please be patient as renovations ensue. :D


    Increased capabilities and Dev presence:

    When logging into the QA server, player will be granted 1,000,000 CotOs and 1,000,000 gold to allow purchases from the in game store as well as resource vendors to assist with making testing easier.
    As seen this month with the Inky testing, Devs will actively participate and provide assistance where possible.

    Path Notes:

    I am building a new forum structure within the Feedback and Bugs tree. Since QA is now the Player Test Environment and will remain open to players unless patching/maintenance will be occurring The QA server will have it's own Bug and Feedback forum. These areas will be reset every month as well so only current issues will be present.
    * The QA patch notes will be published in the Main QA forum (no posts from players will be present there, so that will make it easy to find them.)
    * Testing directives will also be posted in this area. (Players will be free to test/use the QA server in any way they see fit, but players that choose to hammer away at bugs/new/upcoming content and features will be rewarded.)

    Monthly Updates:

    * Forums reset:
    When the new release goes live, the QA forums (along with the Live Bug forums) will be flushed. None of the bug posts will be carried over.
    (They will be archived in a player accessible area as has been done in the past, for reference).
    * DB Copy:
    The Database from the live server will be copied over to the QA server as part of the reset process when a new Release goes live.

    and more ...

    There is still more in the works, but not yet completed (including Bug rewards and Bug Brigade). There is a lot to sweep through, and not just in this area; while the team continues to work on improvements and new content, we will also be re-mediating programs, systems and communications. I am sure we will hit some stumbling blocks along the way, but the entire team is dedicated to making things better in all aspects.
     
  9. FrostII

    FrostII Avatar

    Messages:
    4,644
    Likes Received:
    9,356
    Trophy Points:
    153
    Gender:
    Male
    Location:
    Pacific Northwest
    As are we, @Ravalox , as are we !
    Thanks for what is coming (above).... Let's work more effectively TOGETHER ! ;)
     
    Mhtic, CatherineRose, Vero and 2 others like this.
  10. oplek

    oplek Avatar

    Messages:
    963
    Likes Received:
    2,382
    Trophy Points:
    93
    Gender:
    Male
    I'll be the dissenting voice here, I guess.

    I care. My time is valuable to me. Every moment I spend on QA feels like I'm wasting my time... and that's before any issues like bug reports disappearing into a black hole. I owe nothing to the game. It's a product I purchased. If they want my time, it'll need to be compensated.

    Even the people with the best of intentions will eventually burn out, as the futility settles in on them.

    So I'd say that it would be unwise to do away with any kind of compensation or incentive, if they actually want a critical mass of people helping.
     
    Cordelayne, Rinaldi and Akando like this.
  11. Lord Aventine

    Lord Aventine Avatar

    Messages:
    4,031
    Likes Received:
    7,001
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Caladruin
    I observe that two of four items in the opening post are about incentives. If incentives are important, should we consider an extra paying job for real incentive? Community bug testing in any game (not just SOTA) was never a source of real incentive. Instead bug hunting provides us something else. Its more about the thrill of the hunt and reporting our kill. That's different than incentive.
     
  12. Datendrache

    Datendrache Avatar

    Messages:
    290
    Likes Received:
    1,314
    Trophy Points:
    40
    Gender:
    Male
    I'll join you here, I don't think @Ravalox answered the question about why we're still offering an incentives program and why nothing is being given out.

    I'll make my point clear: the Testers aren't here to test how much abuse they can take, they are here to test the game.

    In a way, testing is part of the game. With the raffle, the odds of winning, the items you receive... Its a part of the game that makes players feel special. Its a part of the game that can take a player from being a normal player to an advanced player. It's was once a way to receive a badge of honor title.

    Portalarium and Catnip know the history. Let's start with this:

    https://www.engadget.com/2007/12/05/garriott-says-too-many-beta-testers-hurt-tabula-rasa/

    Apparently, our pain started there. But its response was baked into Shroud day 1:

    1) They made the QA server originally to be available only to pledges and bundles that had the Developer accounts on them.
    2) They made two sets of rewards, one for Dev+ people as a special perk and another set for regular players.
    3) Player had to pay a healthy sum of money to get access to these "special features" so they could even get access to these rewards. In a nutshell, we were paying Portalarium to be testers.​

    Let that sink in a bit -- "people PAID to be testers to Portalarium." Because what happened next blew my mind.
    • Apparently, the testing rewards were not being distributed properly. Some people got titles and no rewards, for example.
    • I could point a few fingers, but ultimately @DarkStarr was the person that axed the program, -- RETROACTIVELY -- cancelling rewards for two months prior. Some players made Master level during this time, there were probably 3-4 hundred rewards to be given out, if not more.
    • Keep in mind, there were hundreds of bugs reported by players who paid for access to QA, doing nothing more than following Portalarium's directions. These people had incentive and Portalarium brought down the hammer. They claimed exploitation and all sorts of unfairness to them, citing the quality of bug reports and that we all had to suffer equally.
    QA was opened up to everyone, and despite how predictable this was, testing quantity and quality dropped like a stone off a cliff. So the incentives returned with a mandatory post and 4 hours of testing time, as well as a limited subset of the testing rewards program given randomly where the "Dev+" items became inaccessible to everyone (yes... even to those that paid for access to them...)

    Somewhere in all this the Bug Brigade was formed, which was an attempt to organize the community by giving Jira access to some people. There wasn't a lot of details why this got ... dissolved (? still a few artifacts of those days lurking around ?) but it appears to have ended with the QA Team's dissolution. However, this was not a team that was available to anyone except invitation only, and it isn't clear if they received any incentives except a forum "Bug Brigade" title. To the best of my knowledge, no bug tester or bug brigade member ever received an OBNE or special recognition for their efforts.

    Those four hours can feel really long when most of the releases for the last year were so small. Best I can tell they kept with this for awhile, although some players never received anything. In particular, I haven't received a reward since 2017. The odds of that happening were something like five sigmas (99.999% chance of winning) prior to the middle of last year, when there were several months where there were less participants than there were rewards (100% chance.) If its happening to me, its probably happening to a lot of players.

    We all know the names of players that test, and there's little doubt of their skill or honesty and you don't need a computer to verify if a bug has a "Known" or "Resolved" on the title to see if that player was effective. I've made my own lists of players in as little as 10 minutes just to see what the odds were in the raffle... Even tracked the names for awhile when I realized that the same names were appearing over and over... (and then they stopped showing the list because "people would get angry.")

    This leads to the "data loss" of early 2019, where two months of testing reward data (logs only) were lost in a server upgrade, so Portalarium just nullified the rewards because the 30 or so players that posted bugs were clearly cheating or something and they couldn't verify. On behalf of the community, we thank you for your faith in us, especially when you got the good stuff (the actual reports) and the data in the server was really just being used to track if we played for 4 hours.... :(

    But after that, no rewards were distributed.

    If you have been a devoted tester, you probably know all this, you are one of the few that know the most about the game anyway. And there's plenty of messages that have been posted that say the point of being a tester is to help out the game, not the rewards. Unfortunately, that point of view is compulsory.

    Why this lingers around and Upper Tears gets a love nerfing, I have no idea. I can only conclude that the Catnip believes in some fundamental way that the Testing Program is completely useless, in which case it needs to go away and spare all the players the 4 hours of month suffering. If that's the case, there's really no point about revamping the Testing Program.

    Engineer
     
  13. HannahAlpenglow2

    HannahAlpenglow2 Avatar

    Messages:
    269
    Likes Received:
    984
    Trophy Points:
    28
    Gender:
    Male
    Location:
    USA
    There is something to this, but a lot more that goes beyond QA testing issues. I think for QA testing, the issue with inconsistent rewards, acknowledgement of bugs, etc., comes down to the fact that people continue to test regardless of the outcome. In other words, there are amazing players giving their time to try and make the game better, and will do so pretty much no matter what. When you have a game down to a handful of remaining devs, guess what, that loyalty will be taken advantage of.

    This affects most parts of the game, frankly - some amount of players are willing to commit serious time and money to "feel like a part of the development process". SotA has always been pitched this way to a degree, but clearly has ramped up on this since most of the professional staff have gone. It's a really cool concept, I get why people like it, but I do think a lot of folks miss the downside of it - whether through benign neglect or otherwise, people offering to do stuff for free will be taken advantage of. And the game, to outsiders, looks and feels like a fan project, because well, we the fans actually did drive so much of what has come to bear.

    I guess where I am going with this is to adjust expectations, for QA and some much more - players need to stop looking at this game like a typical game development process; it is a super small dev team being heavily supplemented by players, so honestly it is more like a fan run project at this point. Expecting robust QA, features and schedules to be met in a timely fashion - probably time to let that go. Catnip can revamp portions of the current QA process, but without an actual, professional QA staff, it is not likely to matter.

    Sorry for being a grumpy gus this morning and this QA topic leading to a tangent - but when I keep reading how the direction of SotA is to put the power and direction in the players' hands, and that this will lead to new players coming in, I just shake my head. When a material amount of professional developers get hired again, I will believe this game wants new players. Until then, it's about creating a nice space for the current players, which is great for those currently happy with the game world and population I guess.
     
    Rinaldi, Echondas and Datendrache like this.
  14. Anpu

    Anpu Avatar

    Messages:
    3,724
    Likes Received:
    4,161
    Trophy Points:
    153
    I personally spend a vast significant amount of my free time, specifically to report things that I feel will improve and make the game better. And other things as well.

    I don’t do this for a reward (I once won that bug painting set), I do it because I actually care, and want to make this a better game.

    I do however get that some people feel the need to be given a reward for their time (I’m pretty sure everyone likes free stuff ;) ). And that is good too, and does encourage more people to go to qa and enter a prize drawing and test stuff.
     
    Lesni, One Zero, Rinaldi and 2 others like this.
  15. Echondas

    Echondas Bug Brigade – Bug Hunter

    Messages:
    1,952
    Likes Received:
    2,535
    Trophy Points:
    125
    Gender:
    Male
    Location:
    NY
    @Ravalox Please carryover open bugs that haven't been fixed into the QA and Live bug forums when they are reset between releases - or maintain a 'known issues' list that is constantly up-to-date so bug reporters can reference something to avoid making us post the same bugs over and over until they get noticed or fixed or we realize it's a dupe, etc.

    Thanks!
     
  16. Girlsname

    Girlsname Avatar

    Messages:
    229
    Likes Received:
    450
    Trophy Points:
    18
    Location:
    Oregon
    At this point I just “bug test” on live...
     
    Barugon likes this.
  17. Jefe

    Jefe Avatar

    Messages:
    773
    Likes Received:
    883
    Trophy Points:
    93
    So as a thought, the June 19 login reward made xp award from bag a thing. Maybe a bug bounty via xp rewards would be something to consider.
     
  18. Vladamir Begemot

    Vladamir Begemot Avatar

    Messages:
    6,058
    Likes Received:
    11,620
    Trophy Points:
    153
    Gender:
    Male
    The bug testing rewards are awesome. Please bring them back.

    If I see bugs getting answered regularly, I'll start submitting on live again too. I'll throw out an occasional test bug now and then to check.
     
  19. Xiones

    Xiones Avatar

    Messages:
    64
    Likes Received:
    94
    Trophy Points:
    8
    Location:
    Dark Side of the Moon
    Well it is QA testing.......:rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes:
     
  20. Mylady ‘Gin Gin Darlin

    Mylady ‘Gin Gin Darlin Avatar

    Messages:
    85
    Likes Received:
    249
    Trophy Points:
    8
    Location:
    UK
    More like a ‘pre-release public play test server’ or ‘server II’. But that’s more of a mouthful. But yes, I never quite understood naming it a QA server. Quality Assurance? More like Quality Control surely or TS aka Test Server. But anyway. Ho hum.

    @Ravalox does the QA server get wiped before each new release ‘test phase’? Or is it essential just a second ‘live’ server?

    I would expect a test server (which should be a server just for pre release testing purposes) to have regular server wipes, character resets and roll backs and auto levelling to suit the requirements of the current release.

    On that note I think it’s wrong for players to think of and treat it as a second server they can continually play on as some seem to do. It’s not another live server. It’s a testing server after all.

    As for rewards, I know many will frown at this, but imo I don’t think there should be rewards.

    If people want to bug hunt and test and waste their own time doing so that’s their choice. Personally I don’t see why they feel they should demand a reward for it, even if they have ‘wasted precious time’ bug hunting. If folk truly cared about helping the game and bug hunting, then rewards should’t be necessary and time spent wouldn’t matter. Just testing and helping squash bugs should be the reward in of itself.

    And if rewards are to be a thing again they should at least either be tied to the test server only and not usable on live, or something that can be given on live that gives absolutely no advantage, like a plain no frills bound title for example.