My Random Numbers Test Application results

Discussion in 'General Discussion' started by Moussaka, Aug 9, 2017.

Thread Status:
Not open for further replies.
  1. Moussaka

    Moussaka Avatar

    Messages:
    145
    Likes Received:
    243
    Trophy Points:
    18
    Location:
    Switzerland
    Unfortunately, the "Fix your random number generator" thread has been closed, so I have to create a new one since I think the discussion is not quite finished yet.

    I created a small Random Number Generator Test application to actually visualize the "Randomness" of different Random Number Generator functions.

    Here are the results:


    The Test-Application is generating 13 random numbers (from 0 to 12) 10000 times

    VBMath.RND has a problem where the minimum number (0) only shows up about half of the times the other numbers show up, otherwise a nice even appearance of all other numbers

    System.Random (this is the one SotA is currently using) has the problem that the maximum number (12) does not show up at all, chance of occurrence differs greatly between all numbers

    System.Cryptography does produce a nice even appearance of all 13 numbers

    @Chris: Hope this helps

    Cheers
    Moussaka
     
    Last edited: Aug 9, 2017
  2. Waxillium

    Waxillium Avatar

    Messages:
    3,311
    Likes Received:
    9,043
    Trophy Points:
    165
    Gender:
    Male
    Location:
    Rift's End
  3. Randbo

    Randbo Avatar

    Messages:
    164
    Likes Received:
    289
    Trophy Points:
    18
    crypt sure looks like a better rng.
     
  4. jiirc

    jiirc Avatar

    Messages:
    2,853
    Likes Received:
    2,893
    Trophy Points:
    153
    @Moussaka, it looks like you are generating integers. A quick look at how the RNG works in Unity would tell you that the upper range value for the method is used only as a bounds, so 12 is never generated as a value.

    Just an FYI.
     
  5. Gix

    Gix Avatar

    Messages:
    2,203
    Likes Received:
    4,014
    Trophy Points:
    153
    Yes but he's generating 13 numbers... so 12 should be part of the output.
     
  6. yarnevk

    yarnevk Avatar

    Messages:
    402
    Likes Received:
    804
    Trophy Points:
    43
    No it means to get 12 you have to specify the range as 0 to 13, since it is rounding down the real numbers (12.6 is in the range under 13 and rounds down to 12) Of course SOTA could have that same implementation error but that would mean never critting which is easy to debug.

    Programmers count from 0, humans count from 1, therefore programmers are not human :p

    The VB generator is a rounding error, rather than rounding down it is rounding to nearest - thus 0-0.5 happens half as often as 0.5 to 1.5,

    What is more interesting is the periodic frequency pattern on System.Random, every third number has greater chances. In a game where the goal is to meet a DC that is not much of an issue - if the goal is to roll 8 or better it does not matter if 2's and 5's are hotter than 1's and 4's what matter is chances of rolling o-7 vs. 8-12

    Uniform distribution is just one test of randomness. The other is the distribution of hot/cold streaks in smaller sequences of larger runs. True randomness has streaks, while people on the loosing end hate it, that is balanced by people on the winning end liking it. Gambling would not be so successful if losing streaks was not balanced by winning streaks and was more predictable.

    Adding memory to randomness to make things predictable and remove streaks is not randomness by definition. Might as well just use a memory system rather than random numbers that says every 10th one will be a success. Or just make every one a success and just require 10x the mats to make and give 10x the XP.
     
    Last edited: Aug 9, 2017
  7. jiirc

    jiirc Avatar

    Messages:
    2,853
    Likes Received:
    2,893
    Trophy Points:
    153
    As @yarnevk metntions, but more explicitly, System.Random's Next method is called as Next((int1, int2). This generates random numbers from int1 (inclusive) to int2 (exclusive). So Next(0, 12) generates intergers from 0 to 11 while Next (0,13) generates integers from 0 to 12. The assumption here is that the call was from 0 to 12 so 12 was never generated.
     
  8. Barugon

    Barugon Avatar

    Messages:
    15,678
    Likes Received:
    24,293
    Trophy Points:
    153
    Gender:
    Male
    If you're going to call bullshit then the method you use better be bullet-proof.
     
    Black Tortoise and Daigoji Gai like this.
  9. Sycholic

    Sycholic Avatar

    Messages:
    33
    Likes Received:
    27
    Trophy Points:
    8
    Gender:
    Male
    Location:
    Knight's Watch
    I cant confirm nor deny it happens but when it happens to me I believe in bad luck :) .. Im on the fence with this but I know its a hot topic so...
    Anyone who is even remotely talking on this subject should already know that computers really dont generate good random numbers for die rolls, its just how it is. Given I dont know the internal workings of the game if this is somewhat used or not, but could using a quasirandom generator help? https://en.wikipedia.org/wiki/Low-discrepancy_sequence
    Its form of a RNG that eliminates the tendency of computer RNGs to create pattern of it's value results over time.
     
    Last edited: Aug 10, 2017
    Spartus likes this.
  10. Spartus

    Spartus Avatar

    Messages:
    80
    Likes Received:
    184
    Trophy Points:
    8
    Gender:
    Male
    Location:
    St. Louis, Missouri
    I can't confirm or deny it happens either, as I don't play the game very often, but this is a fascinating topic, and has a lot of philosophical implications that have been discussed for decades/centuries (especially what we define as order and disorder), but personally, I've never run into a problem with modern pseudo-random generation for things likes games before. I stick to things like /dev/urandom in Linux and haven't used anything from Microsoft, but the complexity of even simple iterated functions can appear totally random, even to our statistical tests, as many deterministic, dynamical systems exhibit chaos (which is to date, mathematically unpredictable).

    I take advantage of this when using embedded devices that both lack a hardware RNG and also lack a pseudo-random software function (being too large to fit in available memory). I like use the (not cryptographically secure) Linear Congruential Generator. It's old, and I don't know if it meets the low-discrepancy definition that @Sycholic mentioned, but it's a simple, recursive, iterated function, and if you seed it with rotating/modulus user-input timestamps at a frequency that exceeds our nerve response (which is easy for fast clock signals, but pretty much random to humans, since the body is a complex system in itself) then I personally can't tell the difference. "True" randomness is hard to obtain (if such a thing even exists) but you can do things like measure the phase difference/jitter between the independent hardware clocks, or other things like that, but it is much slower. Even /dev/random (the hardware generator) is pretty slow. Obtaining a fast source of true numerical randomness has historically been a difficult thing to do.

    I went and read the entire Fix your random number generator thread that @Moussaka mentioned, and @Ancev asked something interesting:

    ... I 'm pretty sure he meant virtual astronomy, but it reminded me of the recent article that mentioned physicists looking at real astronomy as a way to get bulk random numbers:

    https://phys.org/news/2017-05-stars-random-foundations-physics.html

    And the Princeton Global Consciousness Project, brings into question whether randomness is real or shaped by humans.

    I do have to agree with Chris' last post on this, where he said:

    There is some truth to all of this, as we really don't know much about this topic--in fact we don't even agree on what statistics even is or if randomness can even be measured... is statistics a science, math, art? It uses math, but there is something eerie about it--what is this underlying structure the tool is showing us? Is it real? Is it a mirror of ourselves? Or is it an artifact of the tool? Just look at the frequentist vs. bayesian debates which run up against ontological questions of freewill/quantum physics.

    For cryptography, contrary to many people's assumptions, you don't even need a source of random numbers to be information-theoretic secure, just the word "didgeridoo", for example, will suffice if you assign it a different meaning than is intended. It seems to all come back to human choice and how we decide to demarcate/categorize the world to begin with, so games just need to be really good illusions.

    I prefer games that rely on complexity and simulation (dynamic, malleable, with multiple interacting elements) which can solve most problems without needing many random numbers at all, if you rely on values derived from that emergent complexity. In other words, if game was its own generator, we wouldn't have this discussion. The real world is full of things like this--values that ebb and flow and seem random (and are not predictable by humans), yet they are mostly deterministic systems shaped by humans...
     
    Black Tortoise likes this.
  11. Tetsu Nevara

    Tetsu Nevara Avatar

    Messages:
    405
    Likes Received:
    845
    Trophy Points:
    43
    Gender:
    Male
    Location:
    Germany
    Good work, thanks for the tests
     
  12. Gix

    Gix Avatar

    Messages:
    2,203
    Likes Received:
    4,014
    Trophy Points:
    153
    You guys are thinking too deeply in the matter.

    This is a video game. It doesn't need to have "the most perfect" random generator where each values need to eventually even out. The math doesn't need to be right; just needs to feel right. Just like AI in games doesn't need to have a conscience, it just needs to pretend.

    So when people say "I hate this! It said I had a 90% chance of success and I still failed", you can't just go "that's how RNG works baby, deal with it". As a designer, you still need to make it fun.
    • How hard are your failures? Is it a "all or nothing" roll or is there a gray area? What does the player have to lose?
    • How often can the player redo the roll? If he fights a monster and his 90% crit chance fails, when's the next time he gets to attack? If he kills a monsters and it doesn't drop good items, when's the next time he gets to kill another monster?
    • Is the player actually interacting with the game when the rolls happen or is he/she just part of another form of lottery where they have to sit and wait for results to happen?
    RNG for crafting feels differently than RNG for combat; even if you're using the same number generator. RNG for loot feels differently when you're getting a random amount of gold compared to expecting a quest item to drop.

    THIS is the important parts, folks. THAT is what needs to be addressed. How the CPU comes up with numbers is irrelevant.
     
    Last edited: Aug 10, 2017
    Preachyr, bigta1982, Wextel and 4 others like this.
  13. GrayFog

    GrayFog Avatar

    Messages:
    674
    Likes Received:
    1,400
    Trophy Points:
    93
    Last edited: Aug 10, 2017
    Spartus likes this.
  14. Barugon

    Barugon Avatar

    Messages:
    15,678
    Likes Received:
    24,293
    Trophy Points:
    153
    Gender:
    Male
    Exactly. The issue here isn't with how the numbers are being generated. The problem is that when you fail, your items are destroyed.

    I understand that sucks but many people here seem to feel that they should just be able to masterwork and enchant on a production line. However, that would make super high-end items that use special materials rather common. There would be nothing special about those items because everyone and their brother could easily craft them.
     
  15. Sycholic

    Sycholic Avatar

    Messages:
    33
    Likes Received:
    27
    Trophy Points:
    8
    Gender:
    Male
    Location:
    Knight's Watch
    You wouldn't want that anyway. No-one is referring to cryptography and that's the only time you need true RNG so you cant reproduce its results... by reverse engineering.

    There is no need for perfect RNG, which is why I suggested what I did because its a distributed RNG. It doesn't pattern repeat its results. ie. you not rolling 3 20's back to back nor will you roll 3 1's back to back either... what I suggested is similar to the Monte Carlo methods. I could get all philosophical as well and point out the universe is nothing but one big math calculation and that technically nothing is random... but ya know that's too deep for this topic and moot regarding the game....

    Again remember players (or at least I would like to believe...) are looking for a RNG that distributes it's result values so it seems balanced and fair with its results. So you dont get for example. roll 3 fails on a 95% success rate is 8000 to 1 (95% is a d20 roll technically) and I think people are saying that this happens 'too much' which I somewhat agree upon. Based on what I read about normal computer RNG's this is what it tends to do and thats why casino's dont use normal computer generated RNG either...
     
    Last edited: Aug 10, 2017
    Spartus likes this.
  16. Gix

    Gix Avatar

    Messages:
    2,203
    Likes Received:
    4,014
    Trophy Points:
    153
    @Sycholic you kinda missed the entire point of my post... I'm saying the RNG generator is irrelevant regardless of what "perfect" means to you (whenever that's purely random or outputs outcomes perfectly evenly). What they need to work on is how the output of that RNG affects game-play.

    Players just want to have fun. That's why they react negatively towards failing on a 90%+ roll.

    Right, although I think the issue is more that the player isn't doing anything except looking at a progress bar while crafting and THEN lose items (and time).

    World of Warcraft (to use a tired example) has an RNG in their crafting as well. Except that BECAUSE it makes the player wait idle for the thing to happen, the RNG is used to determined whenever or not the crafting kill got XP or not.

    Elder Scrolls Online (to my knowledge) has a proc that increases the quantity or quality of the item on a RNG. So, like WoW, you're guaranteed something but you have a chance to get something better.

    GuildWars two, aside from its fantastic discovery system, works in a similar way.

    The recurring theme here is that it doesn't waste your time.

    Final Fantasy 14's crafting involves a lot of small lotteries. As the player is crafting the item, he/she has multiple chances make the decision to back out with some results or keep pushing for a riskier, higher quality item.

    If the devs wanted crafted items to be more scarce, then increasing the amount of resources it takes to craft (and controller their rarity) is possibly one of the most sensible ways to accomplish this. This way, the player has some level of control and, because not everyone can access those resources, it further drives the market.

    Despite me not being a fan of economy and crafting I understand that losing materials while crafting is something that can cultivate a decent market for the players to partake in. However, if you're trying to craft a mithril sword and fail, why are you getting nothing for that effort? Why can't you get a lower quality sword that you can try to sell to make some profit out of it? Is your blacksmith character THAT incompetent that, when he fails to craft a sword, he completely destroys it?

    The RNG still happens, but the outcome is significantly different.

    Heck, it'd be more interesting and less frustrating for people if the crafting process was more involved like a mini-game. Sure, you might fail a 90% roll but at least your success would've been influenced by your actions. Like a 10% success rate would make the mini-game super hard to do but a 90% would make it a lot easier.

    I personally enjoy using Shufflebag systems to even out RNG outputs from my games but I know that, at the end of the day, the RNG output doesn't matter. You can't guess what number will come out and that's really what matters in the end.

    This is the kind of thinking that needs to be applied throughout the entire game for all of the different systems that involves (but not exclusive to) RNG.
     
    Preachyr and Spartus like this.
  17. yarnevk

    yarnevk Avatar

    Messages:
    402
    Likes Received:
    804
    Trophy Points:
    43
    Does not matter for an MMO for two reasons:

    Since you roll against DC so seeing a distribution pattern of some numbers being hot are cold are not relevant. You are not betting on the roll being a 4 and worried about it having a pattern that results in it being a bit off uniform. You are betting on the roll being an 8-12 which means the hot numbers balance out the cold numbers because it is the sum of probablities across the range that matters, not the individual probability.

    The other reason is that a single player never sees the distribution themselves, instead they see a random subsampling because all of the rolls are being randomly distributed across players and time and the law of averages does not apply to small samples. It is the random sampling itself that can make even patterned numbers to be random! Even though this simulation is flawed with improper boundary conditions, it did way more rolls than will every happen in your gaming lifetime, you will never see those results yourself.

    Even if a PNRG has a repeatable pattern that goes 4, 8, 3, 12, 1 every 2047th roll - you will never see that sequence. The only way any one could possibly even see these patterns is to have a cheat bot that denies service to other players and captures all of the rolls for themselves. This is why Chris is the only one that has the data to argue statistical validity and why he has posted he is only interested in testcases where you think the displayed number is wrong and not representing the roll because he has checked the roll stats.

    Now certainly there is pros/cons argument on how the psychology of random numbers is fun vs. annoying in crafting and combat systems, but those arguments have nothing to do with the science of how the numbers are generated and tested.

    D&D has survived for decades using random numbers and it is predated by thousands of years of dice games. Even on roll20 who uses truly random numbers (sampled from electronic white noise) instead of dice or pseudorandom because people made similar the dice are wrong arguments, still has gamers raging that the dice are hot and cold. Guess what they are supposed to be because that is what randomness in fact is.

    The only way you will make progress on SOTA about the RNG is how to make the use of the RNG more fun risk vs. reward as gix is explaining, it is not about how the RNG works.
     
    Last edited: Aug 10, 2017
    Preachyr, mute209 and Spartus like this.
  18. Spartus

    Spartus Avatar

    Messages:
    80
    Likes Received:
    184
    Trophy Points:
    8
    Gender:
    Male
    Location:
    St. Louis, Missouri
    I understand, and for the most part agree, with what @Sycholic, @Gix, and others have said, that it's not necessarily the purity of the RNG that matters to make a fun game, which is one reason I mentioned examples of avoiding an RNG altogether. I like all of the comments in this thread.

    I get your point, but just having a discussion of modify X with Y, while they may be technically important, is limiting. Remember, the game is also being described as a "virtual world", arguably the deepest subject imaginable, which is the only reason that I am here in these forums and why I chose to play the game.

    There are many recurring forum threads that discuss real-life animals and whether their appearance or behavior is a suitable illusion, others discuss biomechanics/physics/lighting, some discuss story design, and here we are discussing chance/cause-and-effect in the world, or "luck", not that any of this has to be an exact reproduction of reality. Much of it has to do with our perception of "consistency" (a fair world).

    There is nothing wrong with going outside of science and into the paranormal realm either, as the game is based on this, full of myth, magical powers and objects, and monsters. In my opinion, riding that line between fantasy and fact IS the game. A game, whether deterministic chess or dice-role backgammon, is neither fiction nor fact, it is a fusion of both, an echo of some pieces of reality. How would one implement a good karma system in a virtual world, for example, without delving deep into its paranormal implications? The entire medieval RPG genre is steeped in paranormal territory.

    The simulation hypothesis--it's not too deep--whether or not this is true, the fact that people are "living" in games like SoTA bring it to the forefront. I watch a lot of MMORPG anime (many nested levels of reality which explore such themes) and have wanted more RPG minigames to be added to this game, adding a 2nd nested world to this reality, the point being that my idea of consistency goes beyond physics, graphics, story, and RNG dice-rolls, but also to the beautiful scale-free distribution of design elements, not just visual facades like trees and rocks, but also the ability to use my freewill to simulate within (getting closer to Turing-completeness, for example). This is why I have wanted an open-world sandbox/crafting engine to be added to the game, so we can create our own mini-realities (and even working machines).

    I'm not trying to derail this thread (there is a lot of good discussion here)--going deep doesn't mean going sideways--the PRNG issue cannot properly be discussed without considering the fundamental game (virtual world) design, just like random numbers in the real world cannot properly be discussed without considering the fundamental nature of reality.
     
  19. Sycholic

    Sycholic Avatar

    Messages:
    33
    Likes Received:
    27
    Trophy Points:
    8
    Gender:
    Male
    Location:
    Knight's Watch
    Spartus, the output is exactly what Im referring to... esp for small batch rolls. Again I dont know what they use IG internally. I was just offering a actual 'proven' method that is better for dice/card draw random computer generated results'. Sadly there is no way to avoid RNG even DnD(tm) cant avoid it.... RL or virtual, its a game based on this.

    Gix, def have no problem on when tradeskilling actually is based on player's action also.

    Its a tough subject regardless, because whats 'fair' is sadly too far biased and opinionated based on the individual.... but I can totally get people getting irked when they do like a 12 item batch make, only to have 3 of them fail and then try again and it happens yet again all the while having 17% chance of a critical success which in the end gave ya not even one critical success out of both times (my own personal experience...) Lucky fails only take fuel, but for like enchanting where your weapon gets destroyed.... yeah I can see why people my be screaming about the RNG used now....
     
    Last edited: Aug 11, 2017
  20. Spartus

    Spartus Avatar

    Messages:
    80
    Likes Received:
    184
    Trophy Points:
    8
    Gender:
    Male
    Location:
    St. Louis, Missouri

    It can be minimized by adding more simulation to the game. D&D couldn't avoid it because it didn't have a computer at its disposal and had to limit the calculations to those that a human could perform within a reasonable amount of time. It couldn't perform the needed real-time physics simulations, but videogames are now performing the millions of calculations necessary to do those simulations, some of which result in a complex, causal sequence of events that many players would accept as being realistic. AI behaviors in many games, for example, often appear random, but they aren't dictated by random numbers at every point, the bot is just allowed to roam around and react to a complex, dynamic environment (purely deterministic), and the player suddenly finds themselves in a unique situation that appears random. The better the simulation you add to a game, the less you need to pull from an RNG. Chaos, which I mentioned earlier (a source of deterministic randomness), will emerge in many such complex environments, and, as long as you let user-input influence the simulation at a high-enough resolution, you also have a source of stochastic randomness (from the human body), all without an RNG. So even if you made a fully deterministic game, the user-experience will still be quite random and perhaps even influenced by quantum effects, allowing things like "luck", "karma" to express themselves, if such things actually exist.

    I agree. Even the natural world (or a good simulation) might not be fair by our higher social standards.
     
Thread Status:
Not open for further replies.