Please do more complete testing in QA.

I am a big fan of the game.
I am not going anywhere.
So please hear my feedback for what it is, feedback.
recently @Chris wrote:
Booh, Devotionals broken. Booh, AE's broken in a new and exciting way. Booh, item effects broken. Booh booh booh!

I would be less disappointed if, I saw any testing of these things in QA. Now that the game is officially released, there needs to be a higher standard of release to the production server. With the overwhelming success of the QA stress test, confidence should be high that players will respond and test changes, if you ask them too. Please let us help. We want a good solid game as much as you do. Let us know what you're thinking and want to test, and you will likely get more direct testing of those things. I know I would help, and I play with many others who would as well.

Maybe the team could consider running the QA server all the time. Use it when there are changes to try, and ask for feedback on those changes. Just my .02 gold worth.

Thanks for the hard work, Uncle Sven

 

First did he actually say Booh? If so please paste or quote it.

Testing in QA is for the players to do.

They have tried to give incentives - sadly I see it more as a way players are using it to get a leg up rather than really testing anything.

I have asked over and over for direction or something specific to test but it is ignored.

In the end incentivizing is not the answer but just ask those players who really want to test things out and make the game better.

If chriss really said Booh - - really nothing else needs to be said about it.

 

Those weren't broken on QA. They broke after another patch. You can test all you want, but if it breaks after a fix for something else there is no way they can know until after they release the patch.

 

What Uncle Sven quoted was indeed Chris's exact statement on his "blog" .
Here's the link.

 

I doubt that the devs can always foresee what is going to break, so I don’t know if it is even possible to direct users to test this or that. But a very short explanation on how everybody could help test would be nice even if it only said “go about your regular in-game business for an hour or two and report back if anything seems broken.” I sometimes wonder if most people don’t just log in to try out a new feature or take a look at a new scene because they don’t know what else to do.

 

I spent hours in QA and didn't see these issues. The whole you-tested-it-but-it-was-broken-by-something-tangential-later thing is common in development, particularly for anything that's coupled.

But I will say that I agree that longer QA, plus perhaps a central high-level changelog of mechanisms so we have some direction as to what to look at, would be nice. I tried to get that from the release instructions draft, but some of the things are murky, and have too high a threshold for being in a position to test them... so I'll start with the low-hanging fruit.

(Unfortunately, that also means they couldn't legitimately surprise us with anything)

 

I was also thinking about the possibility of lagging the release by a month... so that we get an entire month to test the QA build, while they work on the next-next release. But I don't know the practical logistics of that. That might help with cutting down patches.

 

No, "Testing in QA" is not for the players to do, it's the developers responsibility. Portalarium has a very small QA team, with only 1 team member who is focused on QA full time. They like to ask interested players in assisting during the latter part of the development cycle, and do provide some small incentives for participation. However, as you said it is unfocused help. Aggressively short, less than 30 day, development cycles with a tiny QA staff means that the release build is very likely to have a lot of bugs. Follow that up with a bunch of quick releases, builds 763-769 for r53 on 4/27/2018, that likely have had minimal QA/QC and you're going to have some problems.

 

You may have a nominally longer QA period, but if you are pushing major functional patches into the build with hours to go before release, it means squat.

 

I agree on having a more specific list of features/bug fixes/new scenes to test released to the QA testers. This would make it a lot easier for the testers to focus on specific points.
Having the development server online 24/7 for testing will just have players playing on QA always as using a second server, I don't think that is a good idea.

To be honest, I love port their monthly content release cycles. Its amazing how fast they are updating. Of course having this bleeding edge code deployments cycles will have a negative effect that bugs will slip in faster then normal. The thing is code is never perfect nor bug free. A lot of times when fixing a bug will introduce other bugs somewhere else in the code.

So making a statement that the testers did not do a good job testing is not really fair, as sometimes the bugs that are being introduced into production were not there in the development server.

What a better solution would be (also with negative effect), when introducing (big) new features/scenes push them to the development server. Open it up to the testers for a few days and tell them to test this specific code/update. The negative effect with this options will be, that development will halt on that specific code push. Meaning they could or work on other updates/bug fixes or fix the incoming intraday bug reports for that specific part of the update.

 

During development there is a limit on how comprehensive you can make high-level automated regression testing, since the game design is in constant flux. That would require an extensive effort to keep the tests up to date as the expected results keep changing.
Now that the game has released, the amount of change in things like skill trees and the math involved should settle down. If that is the case, the creation of macros that use an avatar to exercise combat in the various skill trees should be possible. Automatically monitoring the stats in various scripted encounters should be able to quickly flag when something has been broken.

 

I think the problem is that some of the new release content doesn't get added until the last couple days before the release. Some things get added last minute and by then, most players leave QA thinking they have tested everything already... so some bugs get missed.

Also, sometimes bugs are caught during the QA period, but the fixes for those bugs don't get done until the live builds... but those fixes end up breaking something else by mistake.

I do agree that the QA server should be up all the time and when a new build is created it could be announced for us to test it... and describe what was added so we know what to hone in on.

 

Maybe we need two QA servers?

One for internal testing. And another for us to test on. This gives Port the option of testing code, pushing code to the QA server for us to test on. It would give us a longer testing period. Most of the month basically. Also it allows both groups to keep code testing moving with fewer interruptions.

Thoughts?

 

Basically every software development production environment has this same issue where expected results and business requirements are constantly changing. That's why TDD is so effective. You create your tests first according to the business rules, then you write the software to satisfy those tests. However, the real problem is that it can be very challenging to write meaningful tests.

Couple that with managers not understanding the benefit to writing tests first, and it's obvious why automated regression testing is often left out of non-open source community based projects. Managers simply don't understand the massive benefit to writing tests first unless they've been formally educated on such (which is rare still). Developers can often see it as "busy work" too, but that is more a symptom of developers not having clear requirements before they are asked to do something. That can stem from not knowing what is technically possible as well.

QA automation engineers are a coveted position in successful companies now days. Companies that still aren't hiring QA automation engineers are missing the boat, and left wondering why they can't seem to get out of bug hell. Developers should be writing unit tests first and software second. QA automation engineers should be writing integration tests in parallel with the developers. The entire team should be sitting in meetings discussing business requirements before they write any code.

All too often, though, you see one person on development team spear heading a change without informing everyone else about the new business rule. This ALWAYS leads to unintended side effects. I think that's what we are seeing here, and that's usually a symptom of being undermanned and trying to do too much at once.



And no, it is not the responsibility of the consumer to test the product for the producer. Silly to suggest that we players bear the responsibility of QA.

 

You hit the proverbial "nail on the head" it is exactly what we are seeing here. We all know it, but it's still so frustrating. One can hope that they see it as an issue before it's too late.

 

And that would be great... if this were a AAA studio game. They just don't have the manpower and budget to do it right.

 

I hear this a lot. I actually worked for a AAA studio building an MMORPG. I now work for a team of 10 developers and we have assigned one developer this role. You don't need a massive team to follow these principles.

Sometimes less is more. I think the Portalarium team is working to meet too many goals in each release.