Understanding Probability and Sample Sizes

Discussion in 'General Discussion' started by Poor game design, Nov 5, 2016.

Thread Status:
Not open for further replies.
  1. Burzmali

    Burzmali Avatar

    Messages:
    1,290
    Likes Received:
    1,771
    Trophy Points:
    113
    It isn't like the binomial distribution pdf or cdf is super complex. Take mining for instance, if I mined 400 times with a stated odds of success of 90% and saw 341 successes, I'd be a bit concerned as that has <0.2% chance of happening at 90% odds, but it has at least a 57% chance of happening if the actual odds are less than 85%. You can run the numbers yourself over at http://stattrek.com/m/online-calculator/binomial.aspx

    400 samples isn't good enough for pinning down the true nature of a system, but it's plenty enough to reject statements about a system.
     
    KuBaTRiZeS, Net, Spoon and 1 other person like this.
  2. Drocis the Devious

    Drocis the Devious Avatar

    Messages:
    18,188
    Likes Received:
    35,440
    Trophy Points:
    153
    Gender:
    Male
    How is that any different than taking a coin flip sample of 100 and saying "there's no way we should have 7 tails in a row!"? (per the link I provided there's a 1 in 6 chance of that happening)

    You're taking a sample of 400 and saying the RNG is broken? Really? Ok, well then you must be really confident with your results and expect that eventually you'll be proven correct. If that's the case, more power to you. But I'm guessing everything is fine and you're using statistics incorrectly for the sake of looking like you're winning an argument.
     
  3. UnseenDragon

    UnseenDragon Avatar

    Messages:
    404
    Likes Received:
    1,097
    Trophy Points:
    55
    Gender:
    Male
    Location:
    Columiba, MD
    No, he was saying it's worthy of being looked at. A coin is a generally standard tool which has been tested over centuries and found to be acceptable. Therefore the prior-expectation is of 50/50. A system still in development does not have this, hence why it is reasonable to expect there might be a bug.

    Simple outcome is that statistics are good for finding evidence of problems in a system (#1) or making generalizations (#2). Both are valid uses, and seems (assuming the data collection was okay) valid for #1. Condescending to people for not understanding statistics doesn't help the situation. And for the record, I do have a degree in management science and statistics and teach at the undergraduate and graduate level.
     
    LoneStranger likes this.
  4. Drocis the Devious

    Drocis the Devious Avatar

    Messages:
    18,188
    Likes Received:
    35,440
    Trophy Points:
    153
    Gender:
    Male
    Actually we don't know that this is a new system. Portalarium could be using an RNG that's been around for many years and is used by many games. It could be a default RNG provided by Unity that has presumably been tested A LOT.

    But again it could be broken, I don't know. Neither do you, neither does anyone until it's tested and then by your own logic it might still be broken because the previous tests weren't enough for your liking.

    Let me get this straight? You're saying that it's better for people to not understand statistics and say things like "It failed 3 times in a row at 90%, that can't be right?!" when in reality it could actually be right? You're saying that explaining that is incorrect is condescending?
     
  5. Berek

    Berek Portalarian Emeritus Dev Emeritus

    Messages:
    3,957
    Likes Received:
    12,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Austin, TX
    Let's keep this train neutral please... I think we're getting a bit too dug into the unnecessary bickering of the details. Let's not downplay everyone's intellectual capacity by talking to them like children.
     
  6. Zapatos80

    Zapatos80 Avatar

    Messages:
    538
    Likes Received:
    753
    Trophy Points:
    63
    TBH i'm baffled we're even having this discussion... the ONLY potential issue here (RNG being broken) can only be resolved by Portalarium's huge date. Nothing more needs to be said imo.
     
  7. LoneStranger

    LoneStranger Avatar

    Messages:
    3,023
    Likes Received:
    4,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    Ahh, great, then you agree it should be investigated and that what people are experiencing could be indicative of a problem. I'm glad you finally came around.
     
  8. meadmoon

    meadmoon Avatar

    Messages:
    218
    Likes Received:
    456
    Trophy Points:
    18
    It may be time for me to rest the case I made on page two.
     
  9. Drocis the Devious

    Drocis the Devious Avatar

    Messages:
    18,188
    Likes Received:
    35,440
    Trophy Points:
    153
    Gender:
    Male
    This was said on like the 2nd or third post I made in this thread.
    This was always my stance.
     
  10. LoneStranger

    LoneStranger Avatar

    Messages:
    3,023
    Likes Received:
    4,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    The RNG is only part of the problem. The RNG itself could be perfectly fine, but there are still base values and modifiers and calculations that could be causing the problems with the results. Also, what the UI says may not be what it is testing against.
     
    Spoon likes this.
  11. Drocis the Devious

    Drocis the Devious Avatar

    Messages:
    18,188
    Likes Received:
    35,440
    Trophy Points:
    153
    Gender:
    Male
    Yes, sure. The UI might not be represented correctly. The value modifiers might not be correctly typed in or aligned with the design.

    Here's my question though, lets say that Chris and team go through the trouble of triple checking everything and come back to say that it's all good. It's all working as designed. For the sake of argument let's just say that happens. Now you go to the crafting table and you have a 95% chance of success, but the item is destroyed. You make a few more items and it happens again. Aren't we right back to where we started from?
     
    Snazz likes this.
  12. LoneStranger

    LoneStranger Avatar

    Messages:
    3,023
    Likes Received:
    4,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    Sure, maybe we are back where we started. But.... but what if they find something wrong?
     
  13. Burzmali

    Burzmali Avatar

    Messages:
    1,290
    Likes Received:
    1,771
    Trophy Points:
    113
    Because there are lie, damn lies and statistics, to name drop a well know book on the subject. Just as I have been carefully selecting the numbers I'm using, so are they. The math for runs in a set is a fair bit more complicated, but in this case a fair coin has a 50% chance of coming up heads and the odds of 7 heads in a row is 1 in 128. Those aren't bad odds for a sample of 100 with has 94 potential 7 coin runs. Conversely, the odds of seeing 4 consecutive failures on a 90% mining attempt is 1 in 10,000 and a 400 sample set has 397 potential runs. I could dig up the actual calculation, but it should be obvious that the odds of seeing a 4 failure run in an random 400 mining sample are very small.

    Stats are finicky, add one here and subtract a tenth there and the results are vastly different, especially with larger sample sizes. There is an entire discipline for quantifying that aspect of stats, sensitivity analysis, but you're unlikely to see it mentioned in most non-engineering journals.
     
    LoneStranger likes this.
  14. Isaiah

    Isaiah Avatar

    Messages:
    6,887
    Likes Received:
    8,359
    Trophy Points:
    165
    Gender:
    Male
    I've taken a statistics class before and it was a wonderful experience. I had no clue if I was going to get an A or not, and I really wanted that A and in fact I needed that A for my gpa.

    Fortunately for me I got the A. what I do not know is by what measure he used to determine how I earned the A.

    From what I remember I figured I was just coming short of an A. I may have figured wrong. I may have figured right. regardless of what I figured, the professor gave me an A. I don't know if he gave me the grade because of my test scores adding up to an A, or if he used some sort of grading curve (although there was no mention of a curve and I know nobody else's grade anyway). then again the professor may have just given me the grade. or maybe I was predestined to receive the A regardless what I did. or maybe I aced the final and definitely earned the grade which is starting to come back to my memory that the final was easier than I expected.

    In any case I have no evidence to rule out predestination. really
     
  15. Drocis the Devious

    Drocis the Devious Avatar

    Messages:
    18,188
    Likes Received:
    35,440
    Trophy Points:
    153
    Gender:
    Male
    I'm not suggesting that they can't look at it.

    I'm suggesting that there are people in the forums that are hell bent on them looking at it because they can't imagine a world where 95% success rates could result in so many failures (because they don't understand how statistics work). If Portalraium wants to check the RNG and the UI and the processes that deliver random numbers that's great! (I could even see them saying "yeah we found something it's all fixed now" when in reality they never even looked at it)

    The point is that what's really the issue here is some people just plain hate random numbers and events. If the system is working perfectly its kind of besides the point.
     
    StevieD likes this.
  16. Spoon

    Spoon Avatar

    Messages:
    8,403
    Likes Received:
    23,554
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Sweden
    +1
    This is what I've always considered to be the true culprit. I think the output to the UI text label and the actual roll the algorithm uses differs.

    Which is really what was promised and hinted to with the comments like "affected by the stars" and "location and time of day matters" etc. So I think that the information black hole is somewhat by design, and that there are factors which we are supposed to find out on our own.



    Would have been nice if the /debug would give us some better output on fails/successes and roll outcome, so that we can tally our own larger samples though for Early Access feedback.

    This since for instance what I'm seeing on my end is enough statistically improbable streaks of too many fails or too many successes to be easily dismissed but due to the lack of output formatting I'd never be bothered with collecting the data to be able to prove it since it would be too much work to do so.
     
    KuBaTRiZeS and LoneStranger like this.
  17. LoneStranger

    LoneStranger Avatar

    Messages:
    3,023
    Likes Received:
    4,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    But if there is a problem, then how you can you say that people who notice so many failures weren't on to something?
     
  18. LoneStranger

    LoneStranger Avatar

    Messages:
    3,023
    Likes Received:
    4,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    And I think the failure, in this case, would be to properly convey that other factors are in effect. When it says "95% success rate" and it's not using that exactly, then it's bad feedback to the user and causes people to question what is going on. Unfortunately, if it isn't telling them or at least hinting, it can be frustrating as we know.

    If instead it said "95% base chance" and gave some sign that there were modifiers involved, then at least people know that there is an unknown in effect.
     
    georgegad, StevieD and Spoon like this.
  19. Drocis the Devious

    Drocis the Devious Avatar

    Messages:
    18,188
    Likes Received:
    35,440
    Trophy Points:
    153
    Gender:
    Male
    Because they're basing that on their ignorance. It's like if you go outside and say "it's going to rain today, I just feel it" well there's certainly a chance it's going to rain today, but you're not really using data correctly to come to that conclusion. "You feeling it" isn't based on anything measurable.

    What people should be doing is saying "I don't like how this system works. It says I have a 95% chance of success and then I fail all the time." What they should not be saying is "I counted on my hand the number of times I failed and the math doesn't lie, the RNG is broken!"

    Do you see the difference?
     
  20. LoneStranger

    LoneStranger Avatar

    Messages:
    3,023
    Likes Received:
    4,761
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Petaluma, CA
    I do. I don't say "it's going to rain today, I just feel it" unless I'm basing it on something. For example, the existence of clouds in the sky, their color, the temperature, etc. I don't just randomly say it.

    I don't see people saying "I counted on my hand the number of times I failed" I see people providing numbers and such, even if it seems to be a small sample size, and saying that they think there is a problem. Taking a 10000 foot view of the number of people reporting odd results, even if within the realm of possibility/probability, indicates that it needs to be looked at and I don't mean just at the RNG or the calculations. Working as designed/working as programmed doesn't mean that it's working well or in a fun manner.
     
Thread Status:
Not open for further replies.