We have detected that cookies are not enabled on your browser. Please enable cookies to ensure the proper experience.
Page 1 of 2 1 2 LastLast
Results 1 to 25 of 35
  1. #1
    Join Date
    Aug 2009
    Location
    Landroval
    Posts
    1,166

    Sample size: ONE THOUSAND

    This is basically an extension of the original experiment. I decided to post this in similar style to the other thread about large sample sizes and crit crafting but separately because the sample size is the minimum I'd personally accept for casual testing purposes like this. Will be added to the Lorebook article later.

    HYPOTHESIS:

    Across a time-cumulative large sample (four-digit minimum), application of the optional crit ingredient will result in an expected percentage that aligns with the crafter’s advertised crit probability.

    METHODS & MATERIALS:

    My supreme master Tinker (started at level 43, ended at level 46) tested 1000 jeweller crafting rolls across T1 apprentice through T7 Supreme, during the period of 30JUL2011 through 17SEPT2011. In every single roll, the superior level 35 jeweller tools were used along with a salt of the appropriate tier, giving a final probability of 62% for a crit. The sample was spread across 60 sittings (60 discrete logs), and each individual recorded sitting ranged from a sample size of 2 up to 44; the average individual sample was 17 per sitting.

    The intended sample size was an ideal 2000, but due to a bottleneck of resources (gems and salts specifically cannot be farmed easily), I stopped at 1000. All results were logged by placing the /Standard output channel into its own tab. Results were parsed in Excel 2010 and summarily graphed in pretty colors.

    RESULTS:


    Due to the higher number of sittings compared to the cooking experiment, this was difficult to graph in the usual "100 ratio" style I like, so I had to be a little creative. The row on the very bottom of the graph is the expected 62%; the final actual percent of crit-vs-normal came out to be 62.4%. From the very jagged appearance of the green/blue areas, you can see that streaks and random chance are all over the place, yet the total percent came very close to the expected value.

    Because each sitting on the graph represents different actual sample sizes (the very top row is actually only two rolls, making a sample of 2), keep in mind that the "weight" for every sitting will be different and is not captured effectively in the graph. For example, a sitting of 40 rolls that gives 72.5% is going to have much more weight than a sitting of 2 rolls giving 50%.

    NITTY GRITTY:

    Craft sittings were not consistent--they occurred at various hours of the day and night for over 1.5 months, with each recorded sitting varying from 2 rolls to over 40. I did not interact much with the vendors in this experiment compared to the cooking experiment and did not consistently repair tools (i.e., I forgot). The crafting window was open during every sitting, though I may or may not have had a vendor window open as well.

    Crafting rolls included tokens and gems, which both stacked, and all other manner of usable final products. The crit versions were sold to fund further accumulation of necessary materials (ores, ingots, gems, salts). I didn't keep track of costs compared to the cooking experiment, but I think I came out ahead if not even. The number of rolls per jeweller tier was extremely dependent on the number of salts I had at the time per that tier, and I was unwilling to pay beyond a certain amount per salts at the auction house.

    DEFINITIONS

    • RNG: The computerized random number generator behaves, theoretically, like a die. Even with a known percentage "crit rate," the actual results are still "random."
    • Crit rate: The crit rate indicates the probability for the crafter to produce a critical success result for every individual crafting roll. The crit rate does not apply to a total cumulative sample per se. With an 83% crit rate, it is possible to fail a dozen times in a row; the probability of that happening is low, but not impossible. Please note that I conflate the terms "crit rate" and "crit probability" (the latter is accurate) because a lot of players do that, and I have to be consistent with my previous experiment.
    • Channel (channeling): A timed act with the blue timer bar (such as certain class skills, or mapping, or in this case creating a single craft item); only gems and tokens can be channeled in the jeweller profession.
    • Session: For consistency with my previous experiment, a crafting session is defined here as an uninterrupted series of "make all" channeling. For this jeweller experiment, sessions were mixed into sittings whenever possible to increase the sample size. For practical reasons (limited resources, limited space, etc.), I never did a complete "make all" session that would convert large amounts of material into a single product type.
    • Sitting: A sitting is essentially a single recorded log. Each sitting varied in sample size, though I started out aiming for 10 per sitting. In order to prevent too much log contamination, if I started a recorded craft sitting and realized I needed more of something, I would stop recording (which creates a discrete log) and begin logging again only after I got what I needed. This means that for every login, I could have multiple recorded sittings.
    • Roll: A "roll" here is defined as a single product channeling act that creates a final product(s), with the RNG applying at the end of the channel.


    Additional Discussion:

    I do not expect Rise of Isengard to change the actual underlying critical mechanism of crafting. The only changes that I know of apply to the forthcoming Tier 7, and then mainly apply only to the prospecting (gathering and smelting) aspect.

    I also assume that this experiment applies to all other crafting professions that take crit ingredients. My reasoning is that, from a programming perspective, using different code for different crit objects in different professions would be inefficient; thus, it would make sense that the underlying code behaves the same way across professions, also taking into consideration that the game is built on top of an existing, generic RNG mechanism as well.

    Bottlenecks--salts and gems, but salts were the biggest bottleneck, and I don't have the patience to farm the AH for another 1000 sample. My tinker already had quite a few banked salts and gems because I was not terribly active in her jeweller profession; I might remember to make a few things on a random weekend, basically. But after a certain point, trying to find salts for the tiers where I had an excess of precious metals became a little frustrating.

    Basic streak data

    Crit streaks
    Median: 3
    Mean: 3.266
    Max: 2 streaks of 10 crits

    Noncrit streaks
    Median: 2
    Mean: 2.561
    Max: 2 streaks of 5 noncrits (Huh.)

    Mediafire link for logs.

    WAI.
    THE END. (or is it?)
    Last edited by Trilwych; Sep 19 2011 at 08:56 PM. Reason: Clarification
    [url="https://www.lotro.com/forums/showthread.php?506048-Talent-trees-class-roles-and-player-choice"]Talent trees, class roles, and player choice[/url]
    Crafting crit chance analysis: [URL="https://www.lotro.com/forums/showthread.php?410893-Sample-size-THREE-THOUSAND"]3000 sample size[/URL], [URL="https://www.lotro.com/forums/showthread.php?420198-Sample-size-ONE-THOUSAND"]1000 sample size[/URL]

  2. #2

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Trilwych View Post
    I expect that 1000 is the absolute minimum for an acceptable sample size to test any RNG-based mechanism in LotRO, though 2000 is much more ideal.
    Not a statistician, but...

    Mostly, it depends on what margin of error and level of confidence you're willing to accept. Although I believe the formulas given are for the worst case, so the actual margin of error will be slightly lower. (1000 samples should give a margin of error of +/-2.6% with 90% confidence).

    Hmm... I should try comparing that margin of error with my attempt at finding the margin of error of a given Bernoulli distribution.

  3. #3
    Join Date
    Apr 2007
    Location
    Sitting on a computer chair.
    Posts
    3,093

    Re: Sample size: ONE THOUSAND

    +rep @ OP

    Although it's clear that you work for Turbine and have posted this obvious forgery in an attempt to silence the RNG critics and their attempts to show the world that Turbine uses the world's greatest RNG hack ever developed.

  4. #4

    Re: Sample size: ONE THOUSAND

    I wonder if doing this over so many different sessions would have had any impact on the outcome?
    [charsig=http://lotrosigs.level3.turbine.com/0e21500000001fa17/01004/signature.png]undefined[/charsig]

  5. #5
    Join Date
    Apr 2007
    Location
    Sitting on a computer chair.
    Posts
    3,093

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Resture View Post
    I wonder if doing this over so many different sessions would have had any impact on the outcome?
    Um, no? That's the whole point of the RANDOM number generator. Of course you're more than welcome to run a single session of 1000 attempts and let us know how it works out.

  6. #6
    Join Date
    Apr 2008
    Posts
    3,203

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Resture View Post
    I wonder if doing this over so many different sessions would have had any impact on the outcome?
    That is testable if you can stay online for a very long time, give it a try. "I wonders" are cheap, getting 1000 samples is not, particularly in a single session. Your theory to be tested is that while the OP showed that the RNG is random enough over 1000 samples in different sessions, it wouldn't be random enough over 1000 samples in a single session? I'd personally not think it worth the effort to run the experiment. Of course, I've never been a believer in "the RNG is broken because I had a streak" meme, so I'd not even have done the test that Trilwych did. Fact is, those who are certain in their hearts that the RNG is broken and out to get them won't be swayed by inconvenient data that doesn't support that position...

  7. #7

    Re: Sample size: ONE THOUSAND

    If you were using a L46 character for this, you obviously could not have had Erebrandir's Horseshoe equipped. Therefore, your data is flawed and your analysis is bogus.

    Also, your reliance on facts and statistics, instead of hyperbole and confirmation-bias, is truly disappointing. This is the internet, you know!

  8. #8

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by DelgonTheWise View Post
    I've never been a believer in "the RNG is broken because I had a streak" meme
    Me either, simply because even though I'm not a statistician, I do know enough to realize that streaks are SUPPOSED to happen in a random system. Any computer-generated series of numbers that never contains streaks is actually LESS random than one that does.

    Forced distribution isn't random.
    The forums are not an accurate representation of the thoughts and feelings of the whole player base. Those who like a particular feature are in the game enjoying that feature. Those who don't like it log out to mention it on the forums. It is a relevant but biased source of feedback, and any claims of community desire should take this fact into account.

  9. #9

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by moebius92 View Post
    Not a statistician, but...

    Mostly, it depends on what margin of error and level of confidence you're willing to accept. Although I believe the formulas given are for the worst case, so the actual margin of error will be slightly lower. (1000 samples should give a margin of error of +/-2.6% with 90% confidence).

    Hmm... I should try comparing that margin of error with my attempt at finding the margin of error of a given Bernoulli distribution.
    Yep, most people would be amazed at how small a sample size is really needed. Here's a useful link for determining what sample size you need: http://www.surveysystem.com/sscalc.htm

    Following that you'll see that for an infinite population you can get 95% confidence that your are +/- 5% with only 384 samples, and 99% confidence at 666 samples.

    Still this is good work and I like the chart. It demonstrates the spikes very nicely.
    [charsig=http://lotrosigs.level3.turbine.com/082070000001e0c1c/01005/signature.png]undefined[/charsig]

  10. #10
    Join Date
    Apr 2007
    Location
    Sitting on a computer chair.
    Posts
    3,093

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Ailedra View Post
    Me either, simply because even though I'm not a statistician, I do know enough to realize that streaks are SUPPOSED to happen in a random system.
    Exactly!!! The people at my casino who are trying to "work" the craps and roulette tables are depending on those streaks to make money. Somehow they always seem to lose. Weird how that works. Oh well, keeps me employed!!!

  11. #11
    Join Date
    Feb 2007
    Posts
    12,677

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by DelgonTheWise View Post
    I've never been a believer in "the RNG is broken because I had a streak" meme...
    Me neither, but I'm not sure even a lot of the people who complain about the RNG believe it's "broken". They simply hate that they've experienced an "unlucky" streak. In fact, one poster years ago suggested that Turbine should look for unlucky streaks, and "compensate" for them by delivering a guaranteed "lucky" streak afterwards. Nevermind the problems with that - the point was that what the complainers are asking for is more determinism in their results. Less randomness. Or even no randomness.

    This is a legitimate discussion, but I fall on the side of liking some randomness in my games. If it's deterministic, it feels much less like entertainment to me, and a lot more like "work". If it's deterministic, I know going in that there's no chance I'll get a lucky roll, that it will take X attempts to achieve my goal (where "X" is usually some horrendously large number, since these mechanics are usually just glorified timesinks). I find that so depressing that I'm very tempted to give up before I even begin: I work enough in real life.

    Khafar

  12. #12
    Join Date
    Jan 2007
    Location
    5 Brookbank Street, Troutbottom, Shire, Gladden
    Posts
    833

    Re: Sample size: ONE THOUSAND

    Nicely done. Please start looking into global warming now.
    Pobo is Bad to the Bone. See why here!

  13. #13
    Join Date
    Dec 2007
    Posts
    8,696

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by opurt View Post
    Exactly!!! The people at my casino who are trying to "work" the craps and roulette tables are depending on those streaks to make money. Somehow they always seem to lose. Weird how that works. Oh well, keeps me employed!!!
    It all comes down to people not grasping the concepts of randomness. It's how casinos make money and it's why crafters will swear on a stack of hobbits that there's a bug in the RNG.

    This is why I think Turbine should get rid of the randomness. People don't understand it and will complain bitterly about it. So instead have a cumulative chance of getting a crit. The longer you go without a crit the more likely you will get one until it's guaranteed. That is, fake the numbers and guarantee that if crit chance is 62% that you will get 62 crits out of 100. Cheat and put in history to the rolls. (for instance, take numbers from 1 to 100 and then shuffle uniformly, then deal them out one at a time to determine the win/loss)

    I heard a story once about randomness. Class was split into two groups, one group flipped a coin 100 times and put the results on the chalkboard, the other group wrote down 100 results that they just made up to look random. When the teacher returned she pointed to the set of numbers where they flipped the coins and said "that's the right one".

    The thing is, the odds of getting "hhhhhhhhhh" on coin flips is exactly the same as the odds for "thhthtthht". But people see ten heads in a row and they're amazed.
    Last edited by Lohi; Sep 19 2011 at 06:31 PM.

  14. #14

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by opurt View Post
    Um, no? That's the whole point of the RANDOM number generator. Of course you're more than welcome to run a single session of 1000 attempts and let us know how it works out.
    Perhaps what I was getting at wasn't apparent. It's not actually possible to program for true randomness. Not knowing how Turbine handles their RNG, no way to know if there's a potential pitfall there.

    I distinctly remember some previous maths issues on Turbine's part relating to randomness of initial mob aggro (The "Wi" flag in Asheron's Call).

    Anyway, all that aside it's a pretty impressive test. Doing it on in a single session would take something on the order of three hours just for the crafting time, and there would obviously need to be breaks to offload product and repair
    [charsig=http://lotrosigs.level3.turbine.com/0e21500000001fa17/01004/signature.png]undefined[/charsig]

  15. #15

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Resture View Post
    Perhaps what I was getting at wasn't apparent. It's not actually possible to program for true randomness.
    This is known.

    Sorry, not snarking at you, but every time this conversation comes up, someone will trot out the fact that we're actually talking about a pseduo random number generator, that works from seeds, like it's new information that somehow changes things.

    The fact that a programmatic RNG is actually a PRNG doesn't change anything substantive about Tril's results, as a well-written PRNG will seek only to simulate a truly random distribution over a long period of time, login time and starting seed notwithstanding. It makes no difference in Tril's results if the tests were made in one session, or over many. Put another way, if it were done in one session, and the results were different, that's proof of nothing.

    That's why I made the point I did about streaks....any well-designed PRNG should contain seeds that are streakier than others...otherwise, the system isn't as reflective of a truly random system as it could be.
    The forums are not an accurate representation of the thoughts and feelings of the whole player base. Those who like a particular feature are in the game enjoying that feature. Those who don't like it log out to mention it on the forums. It is a relevant but biased source of feedback, and any claims of community desire should take this fact into account.

  16. #16
    Join Date
    Aug 2009
    Location
    Landroval
    Posts
    1,166

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by moebius92 View Post
    Not a statistician, but...

    Mostly, it depends on what margin of error and level of confidence you're willing to accept. Although I believe the formulas given are for the worst case, so the actual margin of error will be slightly lower. (1000 samples should give a margin of error of +/-2.6% with 90% confidence).

    Hmm... I should try comparing that margin of error with my attempt at finding the margin of error of a given Bernoulli distribution.
    Yes, what I meant to add in the OP is that I wanted a sample of 2000 for a standard error acceptable to me. (I took that out anyway.) I briefly reviewed some of the stat information before doing the hardcore parsing, but promptly forgot. Oh well.

    Quote Originally Posted by FormulaTroll View Post
    Nicely done. Please start looking into global warming now.
    Just let me roll my 100-sided die here...

    I got a critical roll! Does that mean we get hit by an asteroid now?

    Quote Originally Posted by Khafar View Post
    This is a legitimate discussion, but I fall on the side of liking some randomness in my games. If it's deterministic, it feels much less like entertainment to me, and a lot more like "work". If it's deterministic, I know going in that there's no chance I'll get a lucky roll, that it will take X attempts to achieve my goal (where "X" is usually some horrendously large number, since these mechanics are usually just glorified timesinks). I find that so depressing that I'm very tempted to give up before I even begin: I work enough in real life.

    Khafar
    Quote Originally Posted by Lohi View Post
    It all comes down to people not grasping the concepts of randomness. It's how casinos make money and it's why crafters will swear on a stack of hobbits that there's a bug in the RNG.

    This is why I think Turbine should get rid of the randomness. People don't understand it and will complain bitterly about it.
    Looking at the overall crafting mechanisms that Turbine has in place (not including the T7 prospecting changes and whatever), I think it comes down to the psychological expectations of product quality. The reason why I never refer to a regular result as a "failure" is because the yellow-quality products should be quite usable in the game world. And they are--they're better than on-level quest rewards of the same quality as well. Any crits should be viewed as icing, not an absolute right or something (though there have been some good suggestions about increasing crit chances based on 'experience'). A lot of the complaints revolving around determinism--I'm with Khafar on liking some randomness here--also fail to address the fact that guild recipes exist for a reason: the highest quality at 100% crit, in exchange for a cooldown.

    So let's look at the determinism of guild recipes. What if there was a lesser level of guild guarantee-crit but with a shorter cooldown? I believe the scholar guild has a multitude of recipes that use the various rep scroll sizes, but imagine this for something like metalsmith. Simplified, a lesser guild system could look like this:

    You have large, medium, and small guild crests, each with different cooldown lengths.

    Large guild crests should still be limited to the highest level guild products (teal).

    You can consecutively attempt a purple-quality crit version of a regular product using the crit ingredient, understanding that you have a good possibility of a noncrit.

    Or you can sacrifice a medium guild crest in place of the crit ingredient; the product will be purple, but better quality than the crit ingredient version and less than the teal guild product.
    [url="https://www.lotro.com/forums/showthread.php?506048-Talent-trees-class-roles-and-player-choice"]Talent trees, class roles, and player choice[/url]
    Crafting crit chance analysis: [URL="https://www.lotro.com/forums/showthread.php?410893-Sample-size-THREE-THOUSAND"]3000 sample size[/URL], [URL="https://www.lotro.com/forums/showthread.php?420198-Sample-size-ONE-THOUSAND"]1000 sample size[/URL]

  17. #17
    Join Date
    Apr 2007
    Location
    Sitting on a computer chair.
    Posts
    3,093

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Ailedra View Post
    This is known.

    Sorry, not snarking at you, but every time this conversation comes up, someone will trot out the fact that we're actually talking about a pseduo random number generator, that works from seeds, like it's new information that somehow changes things.

    The fact that a programmatic RNG is actually a PRNG doesn't change anything substantive about Tril's results, as a well-written PRNG will seek only to simulate a truly random distribution over a long period of time, login time and starting seed notwithstanding. It makes no difference in Tril's results if the tests were made in one session, or over many. Put another way, if it were done in one session, and the results were different, that's proof of nothing.

    That's why I made the point I did about streaks....any well-designed PRNG should contain seeds that are streakier than others...otherwise, the system isn't as reflective of a truly random system as it could be.
    I would assume the masses don't even realize that a computer generated random number is, in fact, not truely random. Those who do understand how it works know that for all intents and purposes it works perfectly well. Especially for something as meaningless as generating a "random" number between 1 and 100. Those who don't understand, well, they start threads about how Turbine is out to deny them crits until the end of time.

  18. #18

    Re: Sample size: ONE THOUSAND

    This is always an interesting read on when RNGs go bad. Sometimes the doom and gloomers are right. And thanks to the OP for the data.

    http://asheron.wikia.com/wiki/Wi_Flag
    [charsig=http://lotrosigs.level3.turbine.com/0420800000014b10a/signature.png]Ranadin[/charsig]

    Ranadin + Happyfish = Ranafish

  19. #19
    Join Date
    Aug 2009
    Location
    Landroval
    Posts
    1,166

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Resture View Post
    Perhaps what I was getting at wasn't apparent. It's not actually possible to program for true randomness. Not knowing how Turbine handles their RNG, no way to know if there's a potential pitfall there.

    I distinctly remember some previous maths issues on Turbine's part relating to randomness of initial mob aggro (The "Wi" flag in Asheron's Call).

    Anyway, all that aside it's a pretty impressive test. Doing it on in a single session would take something on the order of three hours just for the crafting time, and there would obviously need to be breaks to offload product and repair
    For your first two items, I'll point you to the previous 3000-sample thread that brings up the same topics. That cooking trial had far fewer individual logged sessions over a much shorter period of time. Also, this 'wi flag' business was not limited to a simple RNG calculation, as it was a much more complicated application than "add item to add flat probability percentage on top of RNG," so I'm not sure that's terribly relevant.

    To your last item, though, I wanted to add that there were some practical reasons as to why I wouldn't consider attempting a short-term 1000-plus sample of any crafting profession besides cooking or scholar. For the jeweller trial, I specifically wanted to test all tiers and all types of products, not only the ones that stack. Moreover, the higher tier products have the annoying habit of bringing up the stupid naming window, which slows things down considerably. Since I wanted to test all type of product including nonstacking, the amount of time and space required to sell adds quite a bit more work (listing, sending to storage alts, etc.). And since I wanted the crit sales to fund the trial itself, AH exercises take time and patience.

    But that comes to another point. You don't define what a "session" means to you. If you mean a single "login session," then I don't see how both my previous experiment and this one don't disprove a hypothesis that the RNG differs by login session. In other words, no normal player ever crafts very high sample sizes (four digits) in a single login. The underlying principle of probability here is the Law of Large Numbers--regardless of time or other variables, an actual random results should converge upon the expected value based on sample size. Number of login sessions is irrelevant (or should be, assuming the game's RNG is working properly--which it is).

    This jeweller experiment thus represents highly realistic conditions under which the average player will craft in this game, which is all that matters.
    [url="https://www.lotro.com/forums/showthread.php?506048-Talent-trees-class-roles-and-player-choice"]Talent trees, class roles, and player choice[/url]
    Crafting crit chance analysis: [URL="https://www.lotro.com/forums/showthread.php?410893-Sample-size-THREE-THOUSAND"]3000 sample size[/URL], [URL="https://www.lotro.com/forums/showthread.php?420198-Sample-size-ONE-THOUSAND"]1000 sample size[/URL]

  20. #20
    Join Date
    Jun 2008
    Posts
    1,688

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Valiant_Turtle View Post
    Yep, most people would be amazed at how small a sample size is really needed. Here's a useful link for determining what sample size you need: http://www.surveysystem.com/sscalc.htm

    Following that you'll see that for an infinite population you can get 95% confidence that your are +/- 5% with only 384 samples, and 99% confidence at 666 samples.

    Still this is good work and I like the chart. It demonstrates the spikes very nicely.

    +/- 5% is a pretty big confidence interval though

    i've always been convinced that a RNG is something so simple and primitive nowadays that there's no reason why the one lotro is using should not work
    [CENTER][charsig=http://lotrosigs.level3.turbine.com/0b20c0000000b36d8/signature.png]Slartibart[/charsig]
    "O Captain! My Captain!"[/CENTER]

  21. #21
    Join Date
    Aug 2009
    Location
    Landroval
    Posts
    1,166

    Re: Sample size: ONE THOUSAND

    The standard error for this test was 1.535%.

    To be honest, I'm not sure the usual survey methodology is appropriate, since surveys/polls have a target extrapolated population size (sample -> major city population) and don't have an expected target value (like expecting 50% of people like one political candidate over another). Hum.
    [url="https://www.lotro.com/forums/showthread.php?506048-Talent-trees-class-roles-and-player-choice"]Talent trees, class roles, and player choice[/url]
    Crafting crit chance analysis: [URL="https://www.lotro.com/forums/showthread.php?410893-Sample-size-THREE-THOUSAND"]3000 sample size[/URL], [URL="https://www.lotro.com/forums/showthread.php?420198-Sample-size-ONE-THOUSAND"]1000 sample size[/URL]

  22. #22

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Ailedra View Post
    That's why I made the point I did about streaks....any well-designed PRNG should contain seeds that are streakier than others...otherwise, the system isn't as reflective of a truly random system as it could be.
    I did my own smaller scale experiment last night. I made 100 Journeyman crafting tools, using crit mat for each with a 52% displayed success rate.

    Out of 100 tools which I made, 37 were critical success and 63 were non-crit. While this is a much smaller sample size, it's still large enough to point out that on the smaller scale (which is where most of us operate), the streaks can be very bad.

    Since I can sell a superior tool for twice the price of a normal one, I came out a couple gold under expected for that session.
    [charsig=http://lotrosigs.level3.turbine.com/0e21500000001fa17/01004/signature.png]undefined[/charsig]

  23. #23

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Resture View Post
    Out of 100 tools which I made, 37 were critical success and 63 were non-crit. While this is a much smaller sample size, it's still large enough to point out that on the smaller scale (which is where most of us operate), the streaks can be very bad.
    That's the point you're missing: in a smaller sample size, you SHOULD be seeing "bad" streaks, because real-world "true" random systems have streaks of the same severity.


    If a PRNG system only ever deviates by +/- a few percent, it is actually a poor PRNG, not a superior one. Real life random systems have rare, severe streaks. If you never encountered them in a PRNG, that PRNG does not sufficiently represent randomness. Forcing a system into a narrow range of deviation is no more realistic than forcing it to distribute perfectly...neither is random.

    In fact, there should be a starting-seed scenario with a much lower success rate (maybe only 15% over a 100 iteration sample size)....because that outcome is not only statistically possible, it does happen in the "real world". That's not a "bad seed"...it belongs in the system.
    Last edited by Ailedra; Sep 20 2011 at 04:27 PM.
    The forums are not an accurate representation of the thoughts and feelings of the whole player base. Those who like a particular feature are in the game enjoying that feature. Those who don't like it log out to mention it on the forums. It is a relevant but biased source of feedback, and any claims of community desire should take this fact into account.

  24. #24

    Re: Sample size: ONE THOUSAND

    Quote Originally Posted by Ailedra View Post
    That's the point you're missing: in a smaller sample size, you SHOULD be seeing "bad" streaks, because real-world "true" random systems have streaks of the same severity.


    If a PRNG system only ever deviates by +/- a few percent, it is actually a poor PRNG, not a superior one. Real life random systems have rare, severe streaks. If you never encountered them in a PRNG, that PRNG does not sufficiently represent randomness. Forcing a system into a narrow range of deviation is no more realistic than forcing it to distribute perfectly...neither is random.

    In fact, there should be a starting-seed scenario with a much lower success rate (maybe only 15% over a 100 iteration sample size)....because that outcome is not only statistically possible, it does happen in the "real world". That's not a "bad seed"...it belongs in the system.
    I understand more about statistics than you seem to think I do. My point is simply that crafting operates on a pretty small scale. People get frustrated when they have a 70% chance to crit something and have to try 8 times before they actually get the crit they're going for ... and I'm just saying that it is completely understandable.

    When I make a set of Iron or Bronze crafting tools for a new player, nothing is more frustrating than blowing a bunch of extra mats because I happen to be on a "realistic" bad streak.
    [charsig=http://lotrosigs.level3.turbine.com/0e21500000001fa17/01004/signature.png]undefined[/charsig]

  25. #25
    Join Date
    Oct 2007
    Posts
    422

    Re: Sample size: ONE THOUSAND

    First, let me say to the OP, well done! Thank you for taking the time and expense to compile this data. And let me say that I am completely satisfied with the RNG(s) being used by Turbine for crafting.

    But, I think it's important to point out that there is a difference between global randomness and local randomness. Please see this post I made to an earlier thread on this topic: http://forums.lotro.com/showthread.p...46#post5574246

    For example an algorithm that produced 62 crits in a row, then 38 noncrits, then 62 crits, then 38 noncrits, etc. would pass the test for global randomness that has been presented here. Testing for local randomness is a bit more involved. The problem with testing for local randomness in this instance is that we have no idea how Turbine is using the RNG(s) (how often they seed it, where they seed it from, how many may be running simultaneously, etc.) I personally have not figured out a way to do it. I'm not sure there is one.

    But, even if (as I suspect -- that is purely subjective) there is a some discrepancy in local randomness (too many long streaks in either or both directions would be one clue), it obviously can't be more than a slight discrepancy (nobody is getting 30 crits in a row even though that is technically possible).

    So, all in all I think this subject can be safely put to rest unless somebody has a doctoral thesis they wish explore. The RNG being used by Turbine is globally accurate as advertised.
    Last edited by Mandli; Sep 20 2011 at 05:07 PM.
    [center][size=3][b][url=http://bit.ly/W7Dhy8][color=orange]RIP Aaron Swartz[/color][/url][/b][/size]
    [color=gray]Only The Good Die Young[/color][/center]

 

 
Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

This form's session has expired. You need to reload the page.

Reload