Testing Incentives Program: Updated with more Rare Rewards!

Discussion in 'Announcements' started by DarkStarr, Dec 4, 2018.

  1. jmcclain

    jmcclain Avatar

    Messages:
    59
    Likes Received:
    60
    Trophy Points:
    8
    So if i log into the QA server spend at least 4 hours playing find a bug and fill out a bug report i MIGHT get something for that? ............geee you guys are really going all out for the people that help.
     
    kaeshiva and Mishikal like this.
  2. Duke Gréagóir

    Duke Gréagóir Legend of the Hearth

    Messages:
    4,952
    Likes Received:
    10,666
    Trophy Points:
    165
    Location:
    Dara Brae
    I stopped doing the qa testing as I rarely ever found a bug that someone else did not find Ended up wasting 8-12 hours
     
    kaeshiva and Mishikal like this.
  3. Floors

    Floors Avatar

    Messages:
    4,259
    Likes Received:
    6,604
    Trophy Points:
    165
    Maybe giving a daily buff would help on the real server, like something you get when you buy something. That would encourage more QA testing, since it's a waste of valuable game time.
     
    Mishikal likes this.
  4. Barugon

    Barugon Avatar

    Messages:
    11,197
    Likes Received:
    17,970
    Trophy Points:
    153
    Gender:
    Male
    The biggest drawback to testing on QA is having to forgo character advancement on live. If they made it so that when the new release goes live, you would get the experience earned on QA put into your experience pool on live then that would help alleviate the problem.
     
    Last edited: Mar 4, 2019
  5. Mishikal

    Mishikal Avatar

    Messages:
    1,774
    Likes Received:
    2,388
    Trophy Points:
    113
    Location:
    Brittany Estates
    Yeah, that's the same issue I hit. 4 hours is a lot of dedicated time. I understand they don't want people to just "hit and run" on bug reporting. I don't know what the best solution is.
     
    Barugon and Lord Trady of Blix like this.
  6. Sentinel2

    Sentinel2 Avatar

    Messages:
    798
    Likes Received:
    1,310
    Trophy Points:
    93
    I understand that is a concern. However 4 hours testing on platforms they don't have a lot of could still be useful. Mac and Linux are platforms that don't get a lot of love even today.

    I have a lower and higher end Linux system I usually run my tests on. To help out as much as I can. And when there is a regression, I'm posting bug reports :D
     
    Lord Trady of Blix likes this.
  7. kaeshiva

    kaeshiva Avatar

    Messages:
    2,254
    Likes Received:
    8,607
    Trophy Points:
    153
    Gender:
    Female
    I've pretty much stopped testing on QA. I was never really doing it for "compensation" but, seeing loads of other people get all the shiny things and never getting anything has acted as a bit of disincentive, if I'm honest. Would have been better if there were no "incentives" at all instead of giving "incentives" to some people at random and the rest of you -- too bad. Its kinda sad, cause I know they need the testing help, but seeing other folks run around with titles/cool things for helping that there is apparently no way to earn other than "blind luck lotto" has actually acted as a discouragement.

    Aside from "incentives" - I'd like to see:
    - Make it easier for us to test things (the bug/task list that's been coming with the release notes is a good step in this direction)
    - If you make a design decision that is completely inconsistent with everything in game to date (such as not including certain enchantments on one specific item for <arbitrary reason>), disclose that, we wont waste time bugging it and wondering why it isn't fixed or acknowledged, and you don't have to waste time reading it.
    - Give us test cases. Log in, kill a monster, craft an item, catch a fish, what?

    And if you are going to do incentives, then take the time to do them more reasonably. I know the whole point of using volunteers for testing is that there isn't enough time/resource for testing and you don't want to burden yourself with a testing scheme that requires significant admin time on your part to manage. But the whole rng-pot-luck thing is just bad. Doing it based on how many bugs people submit is also bad, cause you end up seeing dozens of the same bugs every time and that isn't productive or useful of anyone's time to go through. If you're able to track time spent by account, then tie rewards to certain thresholds rather than having it be a diceroll on who gets a reward and who doesn't.
     
    Elas TheWise, Rinaldi and Sentinel2 like this.
  8. Datendrache

    Datendrache Avatar

    Messages:
    253
    Likes Received:
    1,137
    Trophy Points:
    40
    Gender:
    Male
    @Lord British -- if you were ever wondering this is my #1 pet peeve... Thank you for fixing #2.
    @DarkStarr -- Please consider adding something else to the rewards program.

    Expectation: A team of volunteers should be managed enough to get the desired benefit without costing too much of an organization's management time. In this way, the current system is outstanding. The existing bug testing is RNG, the reward count is fixed and probably doesn't take more than an hour a month to complete.

    However, its quite possible this method doesn't encourage volunteers to learn to be better testers. For those that follow startup companies such as Goally, you'd know they follow a simple "reward if you do good" approach which is considered natural progress and direction, used as treatment for Autistic, ADHD, and Elderly as therapy. This was the original testing incentives program, admittedly. And from the spreadsheet below, it appeared to be working.

    The current testing rewards program is like a game you can play as hard as you'd like but you won't be rewarded appropriately, you get rewarded randomly -- but most likely won't be rewarded at all. When the chances of winning go up, Portalarium gets less bug feedback.

    I've been getting feedback from the community that is pitch-black negative against the current program, in fact I don't recall reading anyone's post singing its praises (except for my praise about its management resource control as mentioned above). Not to mention the whole cancellation of the original plan had a bunch of berating to the community for both quality (only one critical reported for a week worth of bug reports) and quantity (people are just trying to get rewards -- see the comment about natural progress and direction) and establishing a system in its place that addressed neither issue.

    What can't be seen from the numbers available to me is how valuable the testing program is. I.e., if its worth Portalarium's time to spend more effort organizing the community to perform more. However, the trending for bug reporting for this current incentives program is now below pre-Persistence levels, and still trending downward. While I'd like to "Fix" this, I'm not sure its a good business case in Portalarium's eyes.

    Anyway, here's the chart.

    [​IMG]
     
    Elas TheWise, Rentier and Dhanas like this.
  9. Alley Oop

    Alley Oop Bug Brigade – Bug Hunter

    Messages:
    8,167
    Likes Received:
    13,262
    Trophy Points:
    153
    what if the rng was kept, but more bugs sweetened the pot for everyone? if you get x qa bugs assigned numbers that month, there's 4 prizes for 500 gold crowns. if you get 2x bugs, 8 prizes for 500 gold crowns. more possible prizes, more chances to win, without making a big administrative burden for port.
     
    Datendrache likes this.
  10. Kelly O'Shay

    Kelly O'Shay Avatar

    Messages:
    347
    Likes Received:
    268
    Trophy Points:
    43
    Great comment @kaeshiva . But to expand on the first and third points, it would be nice to know what really needs to be tested.

    The bug/task list in the spreadsheet with a bug number and a half line explanation of what was fixed doesn't give me enough information to test any of those bugs.
    The bug number should be a live link to the bug so we can read the actual bug text that was submitted. Then, we could actually test to see if the bug has been fixed.

    Or if it is too hard to make the bug spreadsheet have more information, maybe a list of things that need to be tested.
     
  11. Mishikal

    Mishikal Avatar

    Messages:
    1,774
    Likes Received:
    2,388
    Trophy Points:
    113
    Location:
    Brittany Estates
    Unfortunately, Portalarium uses a (IMHO god awful) commercial bug tracker that requires a license for users, rather than one of the many free fully featured and well written OSS bug trackers in the world. Thus they cannot provide access to the tracker without paying out the nose for user licenses.
     
  12. Datendrache

    Datendrache Avatar

    Messages:
    253
    Likes Received:
    1,137
    Trophy Points:
    40
    Gender:
    Male
    Another thought is connected to some code recycling -- namely the Subscription system.

    If the list of people fed into the RNG was fed into a modified version of the Subscription system each month, people can be rewarded as having participated for 1 month. After 3 months, they get a reward, 6 months they get a reward, 9 months they get a reward, etc. This goes on from 500 COTOs (lowest) to Corpion Chair (highest, which might take ~2.5-3 years).

    This accomplishes the following:

    1) Regular and Predictable Reward System
    2) Steady Supply of Bug Reports
    3) Automatic delivery of rewards
    4) Growth of Tester Experience
    5) Value of rewards increases which each drop, keeps the more experienced motivated.
    6) Another Long Term Retention perk

    Because the part of the code that would be recycled here doesn't need a web interface or payment system, it would probably be easy enough to implement. Would 8 hours of development time to recycle, modify and internally implement be worth the potential benefits?

    Sincerely,

    Engineer
     
    Last edited: Mar 5, 2019
    Dhanas, Paladin Michael and Jikininki like this.
  13. Paladin Michael

    Paladin Michael Avatar

    Messages:
    1,968
    Likes Received:
    3,380
    Trophy Points:
    113
    Gender:
    Male
    Location:
    Germany
    It could be simple:

    EVERYBODY who spends time and reports a bug should receive something.

    For example (Amount of Bugs each Release!):
    1 Bug: 50 Gold Crowns or instead: 10,000 xp transfered to our main account.
    5 Bugs: 100 Crowns / 50,000 xp transfered to our main account.
    10 Bugs 250 Crowns / 250,000 xp
    20 Bugs 500 Crowns / 500,000 xp

    A goal for everybody to earn a few Crowns (which encourages more people visiting the Crown store ... and as we know: 50 Crowns is not enough to buy an awesome item ;))

    About special items:
    Let's depend on amount of bugs someone posted over time:
    For example:
    25 Bugs: Antennae
    50 Bugs: Bug Cloak
    75 Bugs: Set of Bug Paintings
    100 Bugs: Bug Sprayer with emote
    150 Bugs: Insect Leg Table
    250 Bugs: Corpion Chair
    500 Bugs: BUG -fighting- PET, a player can use to ride :)
    So a long term goal for every bug hunter would be given :)

    Unfortunately this would need a kind of automatism.
    And I assume there is no time -yet- to manage it ...

    May fortuna be with us :)
     
    Last edited: Mar 10, 2019
    Malimn and Dhanas like this.
  14. Mishikal

    Mishikal Avatar

    Messages:
    1,774
    Likes Received:
    2,388
    Trophy Points:
    113
    Location:
    Brittany Estates
    You know that's literally what the old program was, right? You got rewards based on how many cumulative bugs you reported? They scrapped that program because they'd get dozens of reports for the same bug, because everyone wanted to increase their "bug count".
     
    Jaesun and Alley Oop like this.
  15. Paladin Michael

    Paladin Michael Avatar

    Messages:
    1,968
    Likes Received:
    3,380
    Trophy Points:
    113
    Gender:
    Male
    Location:
    Germany
    This also could be a sign of less bugs overall ;)

    However: I would be happy, if most bugs found on QA could be addressed before / within the following release ...
    Current QA bugs should have priority.

    Another thing I wonder about:
    Writing bugs down (by us and a developer) takes sometimes more time than to correct it directly (for example simple things like typos...).
    It takes minutes to write down a bug report for maybe ONE LETTER or ONE WORD. Two persons are involved.
    And than it is written in an excel sheet and waits for being addressed somewhere in future...

    If a few people ('bug hunter brigade'?) could address typos (and also translations) DIRECTLY, it would be very helpful and effective, I assume :)
    Well, I don't know the IT structure and how this could be integrated ... for sure it's not so easy.
     
  16. Paladin Michael

    Paladin Michael Avatar

    Messages:
    1,968
    Likes Received:
    3,380
    Trophy Points:
    113
    Gender:
    Male
    Location:
    Germany
    Yeah, that's what happens, if people are running wild for special items ;)
    'I Can't get enough' - wasn't it Depeche Mode? ;)
    And unfortunately there is no system which can make people virtuous in their behavior ...

    Yes, you're right.
    Besides:
    If 10% of people would throw in bug reports (double...), what would happen?
    10% more would receive a bug item (earlier).
    Okay. More bug artefacts around ;)

    However: I just see it's really hard to handle for our developers!
    It takes additional time to watch, who was posting how much bugs - and who was the first one if it is posted more often ...

    Yes, it's really not easy to have a kind of fair treatment - that's life :( How wonderful life would be without any bugs ;)
     
  17. Floors

    Floors Avatar

    Messages:
    4,259
    Likes Received:
    6,604
    Trophy Points:
    165
    Is it not possible that just reflect that fact that

    1.) there are less bugs than before AND/OR
    2.) there are less users than before
     
  18. Datendrache

    Datendrache Avatar

    Messages:
    253
    Likes Received:
    1,137
    Trophy Points:
    40
    Gender:
    Male
    Release 44 was the release where the "foot went down", you can see on the chart the sudden plunge.

    On your points, people can find bugs in Shroud all day long. I don't mean this as a blast to the quality of SotA, but its an immense game with changing going on all the time that aren't necessarily announced in detail. If you have an eye for detail, you'll notice them more frequently. Getting that eye requires practice. But its also going to take a time dedication.

    If it were a population issue, it would probably be observable on Steam Charts. However, the population actually leveled off then surged up after the changes were made. https://steamcharts.com/app/326160#All

    If anything, the parachute down was probably slowed by opening up the test server to non-Dev+ accounts which happened right after.

    Engineer
     
  19. Abercrombie_McCloud

    Abercrombie_McCloud Avatar

    Messages:
    103
    Likes Received:
    141
    Trophy Points:
    18
    If I am not mistaken you can upload bugs from an excel ss through an API - but this creates more work for someone to keep track of the sheet and what needs to be uploaded.

    Another comment I would like to make - Every time I took the time to enter something, there was some sort of negativity/attitude about what I entered. Kind of turned me off on taking the time.

    I entered one last release watched it for 30 min - saw someone else enter the same thing (mostly) and it was entered as the Jira ticket...because it was newer...

    Anyway- just my 2 cents
     
    Mishikal likes this.
  20. Mishikal

    Mishikal Avatar

    Messages:
    1,774
    Likes Received:
    2,388
    Trophy Points:
    113
    Location:
    Brittany Estates
    The problem is that end users cannot access and easily search for existing bug reports. I constantly hear in the livestreams that there is already a bug for X, but how is an end user supposed to know? This is why public bug trackers are a good thing (provided they have a method for marking security bugs, etc, as private, which most do).
     
    strabo and Dhanas like this.