Crafting statistics (Gotta love the math here.....)

They've been going on a lot longer than three releases.

 

Percentage = sum of data values / sample size. ( 0 for fail, 1 for success )

Standard Deviation = each value (0 for fail, 1 for success) minus actual mean of success/failure then square each value - take mean of these values (sum of actual minus means squared / sample size)

Use these values to calculate confidence interval (for just the 83% enchant):

Confidence Interval:
±0.21 (21%)

Range for the true population mean:
0.42 (42%) to 0.84 (84%)

Your value of 73.7% (which actually should be 14/22 = 63.3% ) allows for an actual 'true' percentage of 83% within your confidence interval for your sample set of 22.

Interestingly - the game appears to show the top end of the confidence interval as the actual 'target' mean. So although your results are well within tolerance, it does appear that given a larger sample size, you could prove the target mean is not valid.

 

That's what killed high end crafting for me in AA. "RNG" issues that suck all skill and fun out of crafting. Disappointed (not surprised) that SotA is having the same issues. Just like with AA focus on tedious grind and RNG luck does not equal a fun game.

 

NO! Stats nazi sayz you gotta use the binomial distribution when computing probability variance.

Stdev = Sqrt(np(1-p))

Stdev = Sqrt(22 * .36 * (.64)) = 2.25 attempts

2.57 * Stdev gives a 99% confidence interval = 5.78

.36 * 22 = 7.92 successes expected +/- 5.78 = 13.7 to 2.14 successes possible based on number of samples (n).

That's a wide range. Your n too small! You go home now!

But, yes I agree that the RNG does not match expectations; namely; fulfilling the desire for enjoyable game progression and results. Cold streaks can be particularly punishing and frankly, NOT FUN.

 

I'd love to see a crafting system like Vanguard had or ff14 reborn has. Something more than 2 clicks and pray with meaningful decisions.
Or if that's too much I'd rather have the rng in the gathering process (hunt for rare mats) and not in the crafting process. SOME risk and rng is ok, but not the magnitude it is now.

As for the current implementation: The statistics might be ok, but that series of fails which so often occurs is really frustrating. It would be great if the chance to fail in succession would be smaller without changing the total failrate or making it too predictable.

 

Math? MATH?!? NO MORE MATH!!!!!

 

FFXIV has a great interactive crafting design. Can choose to use exceptional mats for +HQ, use skills that push quality over speed of crafting (and vice versa for those longass bulk component runs)

The stability feature was partly RNG driven.

This still feels like a placeholder

 

Oh yeah, we haven't even touched upon the topic of other changes that we feel should be made for crafting such as only being able to craft exceptional by dozens of individual clicks or crafting only 20 items at a time. I'm sure every crafter out there wants this crafting system revamped.

 

The proof of randomness is that improbable sequences exist in larger samples, and that the numbers are NOT uniformly distributed. The only way to get that to happen is to add state memory to the system, which violates the definition of random that future results do not depend on prior results.

H T H T H T H T H T

is as probable as

H H H H H T T T T T

but you never hear from people got five heads in a row, you instead hear the complaints they got 5 tails in a row, reporting bias is remembering only the failures not the successes.

Now certainly

T T T T T T T T T T

is a less probable sequence, but it is not proof that RNG is broken, because that very sequence could be embedded in a larger one that does preserve the overall randomness.

T T T T T T T T T T H H H H H H H H H H

While you may not like the random reality as a user experience, clearly the devs are relying on that so that the crafters do not become predictable such that it is just a matter of time for someone to achieve the percentages, because then the person with the most time and most accounts will win the marketplace because it is predictable. Only the population of crafters as a whole can be predictable, not the individual crafter.

The variance of random numbers means that it is very unlikely for a small sample to achieve the percentages, it requires thousands of rolls to validate the percentages and the sequences are truly random (which does mean clustered sequences exist!)

Only Portalarium can document that the RNG fits the overall odds as well as expectation of sequences, no player can do this because they will only ever be exposed to small samples of tens and even hundreds with much effort. That is not enough to prove the RNG is broken, if you want proof lobby Portalarium to post the server history of rolls.


For a more Dilbert take on this

https://www.random.org/analysis/

http://csrc.nist.gov/groups/ST/toolkit/rng/stats_tests.html

 

Putting the statistical math aside and random number generator concept aside. We're having user experience issue here. Players are being unhappy when the failed on 83% success rate way too many times. Unless there are something else to quell the nerd rage (e.g., add some bias, yield better version of "scraps", reduce items durability instead of just disintegrate it, etc), this issue will just be another straw adding to the back of the camel.

Imagine that you are a patient of some serious disease, the doctor said your surgical procedure had 83% chance of success. You went ahead with it and then became paralyzed or amputee. The explanation of failure as "Well.. 83% success chance is accurate on large set of patients" won't fly too well.

 

Hmmm, if you need to go into the maths this deeply to justify someone not enjoying the experience does it really matter if the 'maths' is perfect or not?

A system that allows this outcome is flawed even if it is working as intended.

Risk versus reward is pointless if you have no way to significantly minimise risk and all you do is cap skills and hope your crafting is in line with your expectations.

 

(Continues stroking his lucky hammer as i scored another 2/3 exceptionals ) lol

 

No one brings math into the conversation to justify someone not enjoying their experience. The math gets brought up because people make erroneous claims based on bad math. If the premise of a thread is that randomness can really suck sometimes and that the game would be more fun with less of it, there would be no math posts. If someone claims that game's RNG is broken because they ran bad on their last crafting batch, that claim is very likely to be refuted, as it should be.

 

Their point at large is the system isn't fun.

They are making the same mistake if bringing math into it tu justify their feelings just like those who claim the math is right and defend the iteration of crafting wet have

 

Random number generator algorithm's suck. And rolling a dice for my chance to lose something sucks more.

 

If someone posts they do not like RNG being used in crafting, that instead they want only every 10th craft to fail in in a predictable manner if 90% is listed, then they need to consider the devs position on the economy that they do not want every crafter with time on their hands to succeed because that leads to those with the most time and accounts 'winning' the game.

I personally do not like the devs position because that is based on the game being a MMO grind when it was sold as a single player story based RPG, because it means you may never get lucky enough to craft things by yourself. The entire point of crafting in Skryim was to make your OP dragonbone armor, no fun if that is denied to you. However, the devs position for the MMO is if you can't make it then fund the game and buy it from them or another player. The RNG is an important part of making that happen by design so that the economy does not break.

Now if you want to say the crafting is broken because it deviated from it's original design direction then I would completely agree, but blaming it on a broken RPG algorithm bad math requires proof beyond your personal anecdotes. The players that do not like the RNG need to propose predictable crafting methods that do not lead to broken economies if they want the devs to consider them, and stop blaming it on broken RNG because nothing will be done.

The RNG is not proven broken because someone has bad run of luck, as all that does is prove it is not predictable and indeed is random.

The only way to prove it is 'broken' is do all the NIST statistical tests I provided in the link on the server logs. I especially like the random B&W bitmap idea because it enables people to see the entire population is random with no pattern even if they can see local runs of good and bad luck.

 

I agree with you that it's a mistake to use flawed math to make unsupported claims in a situation where no math is even necessary to make the desired point, but I don't think there's anything wrong at all with anyone correcting that math and refuting those claims once they are out there. If anything, it's a service, and it's exactly what people are asking for when they make threads with "math" in the title, or "The RNG is broken", or "stats inside". If people want to use results to support their opinions, then it's pretty ridiculous for anyone to expect no scrutiny of those results or the methods used to obtain them.

Even without doing any further reading or performing any math, a person could learn a couple of very useful things from reading this thread and others like it:

1. When analyzing results, a very extreme deviation from expectation is required for the results to have any significance over a very small sample. The larger the sample, the more that results should converge towards expectation.

2. In order to gather meaningful results, you either need to always be gathering results or deciding beforehand what results you will analyze. If you choose what results to look at after the fact, having some knowledge of the results, those results are biased and lose some meaning because you might not have looked at them if they had been different.

Those are great things to know, and if more people gave it some thought, maybe there would be less static and more constructive conversation.

 

@Knightmare2 i feel your pain, my sample sizes are only anecdotal so no need for the staticians to correct me. But 98% success should mean that, and if it did then I would have had more than 1 wand from the 10 exceptional I started off with.

A whole forest = 1 wand is what I get from my current user experience and it pisses me off. Someone said it reminds them of Archeage RNG, agreed, I quit that game for that very reason.

 

It turned out the maths behind critical hits was broken, this was investigated because several people posted screenshots of 'good luck'. The sample size was just as insufficient as the threads we keep seeing about the crafting, yet Chris investigated it anyway and found the problems in the code.

I disagree. The sample size is not large enough to prove the RNG is indeed random.

 

Let me restate what I mean, people are using short sequences to prove the RNG is broken, but lack understanding of statistics that such sequences of good and bad luck must exist for something to be random. Yes it does require large sample sizes to make sure the distribution of patterned sequences is in line with randomness, but only Portalarium can do this.

This is not the same as crits bug, where it was clear it was not an RNG problem because the math and logs clearly showed people would ALWAYS crit in certain cases which is more than just luck.

Again if you want proof the RNG is broken, demand Portalarium post server logs of rolls and that they run the NIST tests of randomness and see if the dice are loaded. That is not going to do anything to change people not like crafting using RNG because Portalarium wants rares to actually be random and not predictable because this is designed as a grindy MMO.

90% does not mean 1 out of 10 will be successful, it instead means that 1,000 out ot 10,000 rolls will be successful, and that those 1,000 successes are distributed amongst the population and not controlled by the ten people that are 100x better at the game because they are retired and play ten accounts each all day, then run their hundred macros all night.

If players do not like that fact of statistics, then suggest a better system that does not RNG that still implements rares without predictability, since that is what Portalarium wants for their MMO. Or complain that they should have stuck to their story based no grinding soloable RPG kickstarter promise.

But don't blame the RNG if you do not understand random statistics.