Understanding Probability and Sample Sizes

Anyone else tired of people thinking they are the only ones who understand stats?

 

It would not improve the user experience to high pass filter the RNG,because it makes the individuals results more predictable. High pass filters can indeed significantly lower the quantities needed to obtain a satisfactory confidence interval.

While you think that is what you want, the result is that the player economy collapses when the retired multiaccount player working the game can achieve satisfactory confidence that their quantities are high enough to be predictable.

When all the frequencies are properly represented in the random sequence, then it is those very streaks of good and bad luck that means the casual player with good luck can win the marketplace against that hardcore player having bad luck. Unless the hardcore player is heavily botting with temp accounts - they can never achieve statistical confidence in their results as long as the RNG is not being filtered.

 

To be fair, there were some amount of people complaining as a result of a poor understanding of how probability works. And then others, who understand how it works, but just don't like it.

This is a good point. I don't mind the current implementation of RNG. I think you could take some measures to give the player a little more agency in crafting though. One thing I considered was perhaps giving an option to save failed enchantments/masterworks. Let's say you have a +7 item that fails, you should be given the option to save that item for the price of 1 COTO per item level (in this case 8...base item + 7). Saving the item should prevent it from being used in any further enchanting or masterworking. This way you can pay a steep price to at least get your item back, but if you are shooting for that +14 item, you have to start all over. I suppose COTO values are a matter of perspective, but I wouldn't spend 8 on something I didn't really value.

 

You are incorrect in your assumptions about confidence interval. In a prior thread I simulated multiple sequences from 20-10,000 long many many times all day at work. It took hundred runs of several thousand long sequences to achieve 5+/-1% for a crit. At hundreds of trials there was still people that failed and succeeded 5x more than they should - the actual crit range was 0-25% not 5%. Anyone that has played D&D with honest dice knows this.

Confidence intervals are about how sure you are that the average roll is fitting the probabilities, that the averages are settling into where they be. But when considering a MMO with people posting their bad results, never discussing their average results, and certainly hiding their good results to exploit a good thing - then you need to luck at the entire range of results and not just the average.

My work has to deal with random audio signals, where one second contains 48000 random numbers, and I generally average for 30s to get confidence in results of not just the averages but the outliers. I also have to deal with customers that think 100 samples having results 5x worse than others is a failure in manufacturing because they also do not understand statistics means those failures will occur even if the averages are only 2x worse, that the guard bands are set accordingly so that millions of samples can be produced, and when they sign up to those guardbands they are going to get samples at the edges because of random statistics.

 

The utter failure here is that some people continue to take the topic of a "RNG" with complete and utter raw statistics and probability. Example: Comparing the crafting algorithms with flipping a coin. They then apply this to what is probably an extremely complex algorithm that our dev's have coded for crafting and then make complete assumption that this the system is correct and attack the vast majority of people who continue to show with real data that it continues to be below the probability.

The fact that people can be so certain about so much they know nothing about (until the dev's say provided us the CODE that determine how say an exceptional is made) should say something here.

 

As a math major, I could contribute to the discussion, but I have always hated statistics.

 

This is a good idea that merits discussion in it's own thread. Having played another game (Wurm Online) with the same crafting economy is that there is no market for middling goods when middling goods are destroyed in the effort to get the rare goods. The only ones that make middling goods are those that sell as throwaways while they are skilling up, and thus they have no selling value. And the only ones that make noob goods are those looking to skim the fresh crops in the starter towns, often mixed in with scammers.

If middling goods have value of paying for the rare good as finished middling goods, they would eventually wear out cannot be COTO repaired and cannot be made rare - then there will be a market for them because there needs to be something between rare and noob goods.

 

Only Portalarium is capable of showing rolls are not fitting probability, because by design RNG means that NO single player can achieve statistical confidence in their results. Saying you failed more than the percent success is not proof of lack of randomness, it is in fact one of the proofs of randomness - randomness means there will be streaks of good and bad luck that are beyond any ability of any one player to overcome. An MMO relies on it by design. If you truly think their rolls are flawed, then your time is better served demanding Portalarium run NIST randonmess tests on their server logs, as no amount of crafting on your part can ever point out it is a problem because a single player cannot feasibly distinguish flawed algorithms from bad luck.

In fact I would guess that even Portalarium cannot do it with server logs, because this is not WOW and population is much too low. Instead they would have to set up a server simulation with populations full of bots crafting from infinite resource.

And even if they did do that test and found that Unity RNG works just fine and that the reason you fail is because of their algorithm on top of that, do you think they would tell you? RG is on record saying your diet, the weather, the time of day, the phase of the moon to all be hidden variables that require player discovery to find. They of course will never say what those algorithms are because then there is no player discovery.

 

I already have asked for this in several threads including the one I posted here earlier that the dev's deleted.

In fact I went further in saying that it probably says something that they have this data and have not published it to the backers after we have asked for it a few times.

 

I think part of his point is that it needs to stay hidden so that players never know how many samples they need to go through to achieve known outcomes.

 

The very first item I enchanted went to +9. I actually looked for info if the statistics were broken and was considering submiting a bug report. It broke next time i attempted, it was giving me silly low likeleyhood but it had been doing so the whole time, my score was as at character gen for a mage and I had honestly thought it would break much earlier. Had I been aware it was not common I would never have chanced the less than 20% to upgrade again, I simply assumed the mechanic was broken due to early release and wondered how far it would go.

 

I know a lot of people agree with you.

I know a number of games tend to hide certain mechanics, but i still dont understand why.
How is it a bad thing for people to know how game mechanics work?
I can see why hiding personal stats adds to character development in some ways, but I cant see how knowledge of functional mechanics like crafting should ever need to be hidden, especially not to testers during game production. If people dont know how the system functions they cant use it effectiveley.

I can understand we hide mechanics from loot chests, but if there are hidden variables in item creation process for time or location like some are suggesting then they need to be publicised.

 

I dont think they should have any issues releasing statistics. For example:

100,000 craftings have occurred on the server since persistence and there was an average chance of 22% to create an exceptional for each crafting and say their average was actually 18%.

They could easily run this stat in a few minutes.

 

I think the thinking is: you don't want to reveal the number of samples it takes to achieve an extremely low margin of error as it could be used to game the system to essentially guarantee certain crafting results (although the number might be so astronomically high that it doesn't matter). What probably would be feasible is for them to say, 'we analyzed all crafting for exceptional outcomes and determined that they are produced at a rate of 25% +/- 0.9%'. That would give you a sense of the overall statistical error in the RNG without revealing actual numbers in the sample. Would have to be restricted to same level crafters, though. But it still wouldn't be directly relevant to individuals as their sample size is so much smaller than the total. Ultimately, though, I don't think they'll ever provide enough data that amounts to anything more than, 'trust us, it's working'.

I think there should be mechanisms in play with the addition of RNG that give players more agency in crafting outcomes, but still make rare stuff, well, 'rare'.

The problem with most games is the attempt to meet halfway between numbers people and immersion people. I'd rather they pick a side and go all the way with it. Either release all the algorithms and provide UI support for them, or hide all the numbers and use qualitative descriptors (e.g. good, better, best) so that everything must be done by 'feel' rather than 'calculation'. Would picking one side really drive away the other side?