it'd certainly save a lot of discussion on this very tired topic.

Well, this is a discussion forum, after all.

I'm sure the randomization code is perfectly fine. And I have no qualms with the randomization of my playlists on the empeg. BUT, this is a distinct anomaly that not only I have noticed. Clustering is one thing - but back-to-back pairing of songs repeatedly over years of use is more than just a false perception on my part, I'm sure of it.

I also stream my playlists randomly through winamp and have not noticed the back-to-back pairing happen at anywhere close to the same frequency.

Anyway, it's just an anomaly I wanted to chime in with.

- trs

it'd certainly save a lot of discussion on this very tired topic.

I don't mean to be a jerk, Tony, but if you don't like the topic of this thread, don't read it.

10-4.

I suppose my first question to you would have to be: how many albums do you have of Collective Soul, Dave Matthews, and Tom Petty? More albums means more of a chance of a back-to-back. Also, some "greatest hits" albums can be longer than others (IIRC, Tom Petty's Greatest Hits has far more than the 12 tracks Tony's been messing with). The other piece of information is how many total tracks are you shuffling when you get this behavior?

Just plaything the program a bit I do notice quite a few back to backs, though this is generally with artists who have more than one album. If you weren't shuffling too many tracks (we've been talking around 3000) this would also change the test. You might just want to play around with it and try plugging in some numbers you think represent your data. Though back-to-backs don't get highlighted, they are still easy to see. Of course, the program doesn't show back-to-backs from multiple artists on the same grid but I still think you might be able to tell what kind of behavior is reasonable.

As for me, I typically don’t see any of this behavior because of how I’ve structured my playlists. In fact, I hardly ever use “shuffling” at all. Instead I have a special playlist that picks a single song at random from 14 artists chosen at random, and then repeats the process a dozen times or so. This effectively circumvents the “clustering” and “back to back” behavior we’ve been talking about.

Seeing Bitt posting reminds me I should probably include a picture of what this looks like now for the non-windows folks.

Clearly, there is some combing in all random selections. What I'm not sure is if the amount of combing showing up in my program is the same as on the Empeg. It's hard to tell because ultimately we are perceiving this data in a different way than listening through the playlists. But here you have it anyway.

Not being content to judge things by eye, I have been running some statistics on actual Empeg generated shuffles.

To accomplish this, I took a virgin Mk IIa and loaded 1000 short (200ms) songs into a single playlist. Setting notify=1 in the config file allowed me to capture the player-generated shuffles off the serial port. And a little scripting in minicom allowed me to put it in an unattended loop.

I actually thought I had found some subtle, but statistically significant, anomalies based on a 30 shuffle sample. But when I tested them on a second 150 shuffle sample they disappeared.

As much as I hate to admit it, I think what we have all seen is really just our human instinct to see patterns even if they aren't really there.

--John

Instead I have a special playlist that picks a single song at random from 14 artists chosen at random, and then repeats the process a dozen times or so.

Wow - how the heck do you do that?

- trs

Wow - how the heck do you do that?

It’s been discussed before and it takes a little bit of time to set up, bet here’s how I’ve got mine set up:

-First you create your "weighted shuffle" playlist and create a single playlist underneath it that has “select X tracks” checked and set to 14 (or whatever number you like. 14 tends to be a solid hour if you don’t skip any songs).
-Copy and paste the new playlist into the “weighted shuffle” playlist a dozen or so times so that it’s repeated about a dozen times.
-Now under the new playlist you just copied create a separate playlist for each artist you like.
-For each artist set the “select 1 track” option so that for each artist that’s selected you’ll only get 1 track at random
-Copy the songs you like into each of the appropriate artist playlists. You can optionally make certain songs come up more often by using similar techniques with sub-playlists at this point.

This has the effect of selecting around a hundred and fifty relatively equally weighted songs. Sometimes you do get two songs by the same artist close together (like the end of one set of 14 and the beginning of the next set), but the more artists you have the more rare this becomes. Even when it does happen, I generally find it’s not undesirable and you don’t here the same artist for at least another 14 songs.

There actually is a way to take this to another level of ridiculousness so as to NEVER get songs by the same artist close together (by making yet another sub level with half of the artists in each) but I’ve found the method outlined above works just fine.

Other people I believe simply create on list with every artist set to select one song per artist, but my method feels more like a “radio station” to me so that’s what I do.

Wow! Thanks for the tip!

- trs

Random song
Random song
Collective Soul song
Collective Soul song
Dave Matthews song
Dave Matthews song
Random song
Tom Petty song
Tom Petty song
Random song
Random song
Random song
...

This really does happen quite a bit

Yup, and if you look at the output of FerretBoy's program, you can see why this is normal and common in a truly random shuffle.

It's all about perception. I was (as you are) perceiving that this is unusual because you're sitting there, in "the moment", hearing two Tom Petty songs in a row. Or three in a group of ten, or whatever. And you're thinking to yourself that's gotta be a mistake. Because all you're hearing is "that moment", that short period of, say, ten songs, while you drive to work. So any statistical clustering is going to sound like a mistake. But what you're not seeing (and what FerretBoy's program clearly shows) is where all the other Tom Petty songs are in the shuffle. And how they releate to all the other songs on the player. And once you see it in that light, suddenly some statistical clustering makes perfect sense in a random sample of that size with the artist/album criteria which makes you consider something a cluster.

See, in order to hear and perceive the true distribution of an artist or an album in the shuffle, you have to listen to the whole shuffle. And with thousands of songs, that would take days. No one ever listens to their player for days at one sitting. You listen to it in short groups, as you're driving, or working. And your perception is colored by that sub-segment of the larger statistical sample.

Although I am far from being a statistician, everything I've heard and read suggests that there is no true random number generator. They are only pseudo random, so there is some pattern to their operation. Hence not even the Empeg is immune, although I'm sure that is not what is causing everyone's aforementioned experiences.

Stu

Yeah - I suppose you're right... I just wanted to go on thinking my peg was haunted.

there is no true random number generator

Radioactive decay works pretty well.

All this aside, explaining why an annoying thing happens makes it no (or very little) less annoying.

It would be nice if a randomization algorithm could be implemented that minimized this sort of thing. Since the empeg is playing each song only once, though, that becomes harder and harder the further you get into the playlist. So maybe trying to do it would make the shuffle seem less repetitive toward the beginning; few people probably listen to a whole-empeg shuffle all the way through, anyway.

(The simple solution I see is to generate a random number modulo the number of songs, discard any number less than, say, 40, and use that as an offset from the currently playing song. Not perfect, but probably closer to it without creating much more CPU overhead.)

everything I've heard and read suggests that there is no true random number generator. They are only pseudo random, so there is some pattern to their operation.

Well, all random number generators nowadays are based on input that's different each time. For example, most use the system time clock as a seed, which is a very large number when you take into account that it includes whatever the smallest increment of time the clock measures (which is going to be different depending on the chip and the code).

Agreed, that two pseudo-random numbers based on the same seed will produce the same output, but when the seed numbers are based on a time clock, (and the start trigger time is based on a human input like pressing the "go" button, or heck, even what time of day they choose to turn on the computer) you'll never be able to predict the output and it'll be different every time.

although I'm sure that is not what is causing everyone's aforementioned experiences.

Right, exactly. Whether the number is truly random or only pseudo-random doesn't apply to our discussion here.

Added the FAQ entry. It'll probably get updated as this discussion continues.

Guys, it is always a great pleasure to read you all. I had skipped this thead completely believing that it was useless to disucss about randomness since whatever pattern you would consider random was instead, as I said, a pattern, and therefore the only way to find out if the empeg was actually randomizing was to get the code.
Instead, this beautiful discussion was going on and I was missing it. Ok, I learned the lesson, never skip a thread on the empeg bbs.

So, just for the sake of the discussion, I think we should remember that randomness does not really exist in the universe. A random output is just the outcome of a number of causes and effects that are uncontrollable and impossible to track and calculate by us poor human beings.
I'm sure you remember the beautiful illustration of the chaos theory in Jurassic Park (book or movie): IF we let a drop of water fall on the back of our hand, there's no way to know on which side it will fall on the ground, unless we calculate the wind, blood pressure, pulses of the nervous system, our emotional status, rotation fo the earth, lights around, and all the billions of factors that determine the reasons why the drop choses this path rarher than another one.
IF we could calculate and compute everyhting, we would get rid of randomness and predict perfectly the outcome. But it is too difficult. we can't . So we have to face and accept "unpredictability" .

My point is that there's no such thing as a random number generator. There is just something that can give us the impression that there is no pattern, which is an illusion in the first place. And, as we could see here, i'st an illusion we tend not to believe in, since e see patterns everywhere!

Ok, that's it with my purely "academic" (and probably wrong) contribution to the discussion. Sorry if I bored you, but was just fascinated by the issue

Added the FAQ entry.

Looks good and should make the empeg guys proud! Speaking of whom, have they been reading this or have they just been ignore it as another "random number generator bashing thread"?

And even if could calculate all those factors, the act of us observing the drop of water changes the outcome.

randomness does not really exist in the universe

What about quantum mechanics?

Well, presumably there are reasons behind random behavior in Quantum Mechanics as well. Problem in that case is, I guess, that what we consider time, and therefore the cause-effect relationship between events, totally needs to be reconsidered, and decidin what comes first and what next is quite meaningless if you can manipulate time as any other dimention... Hope I did not forget the little Quantum Mechanics I studied...

Anyway a Quantumpeg is not a bad Idea... MKIV? ;D

I really wish the empeg fellas would consider opening up just the shuffle algorithm's source code to put this issue to rest.

OK, for the first time only, live on air, we're proud to present the entire of the shuffle algorithm source code! (For the non-custom-shuffle case, anyway.)

Here it is:

std::random_shuffle(running_order.begin() + from, running_order.begin() + to);

Good, isn't it?

Peter

Speaking of whom, have they been reading this or have they just been ignore it as another "random number generator bashing thread"?

Quietly confident, mate, quietly confident.

Peter

std::random_shuffle(running_order.begin() + from, running_order.begin() + to);

There! That oughtta shut up all of those doubters!

OK, for the first time only, live on air, we're proud to present the entire of the shuffle algorithm source code!

Would you believe I did this very thing to one of our more "persistent" users? I actually copied and pasted the source into an email and sent it to him. Last I heard from him too, though this had more to do with massively confusing him than anything else!

Although there were many attempts to postulate 'hidden variables' behind quantum mechanical observations, and Einstein's dislike for the idea of 'God playing dice', current thinking is that the nature of events at atomic level (e.g. polarization of paticles resulting from radioactive decay) is genuinely random.

In reply to:

---

Agreed, that two pseudo-random numbers based on the same seed will produce the same output, but when the seed numbers are based on a time clock, (and the start trigger time is based on a human input like pressing the "go" button, or heck, even what time of day they choose to turn on the computer) you'll never be able to predict the output and it'll be different every time.

---

This may seem like a good idea, but in my line of work, anyone who uses this method to generate a random number is asking for trouble. This is one of the easiest to break. We have a selection of exploits to break TCP sequence numbers so we can hijack sessions - most of which depend on system time as a seed.

See here for an amusing demonstration of the lack of randomness in computers.

This may seem like a good idea, but in my line of work, anyone who uses this method to generate a random number is asking for trouble. This is one of the easiest to break. We have a selection of exploits to break TCP sequence numbers so we can hijack sessions - most of which depend on system time as a seed.

True, but we don't need cryptograhically strong random number generator on Empeg.

Of course, the main trouble with Empeg shuffle is human propensity for pattern recognition at any cost, so to say (take, for example, Rorschach test or constelations).

Another interesting corollary springs to mind - has anyone tried measuring how long it takes for a track by a specified artist to come up (assuming a DDD random shuffle and a reasonable similarity between numbers of albums for each artist?)

The reason I ask is that since February I have not heard a single track by Pantera, although I have 5 Pantera albums on my empeg. I have, however, heard 3 by Madonna and 3 by System of a Down.



I like Ferretboy's special playlist, though. Might take a while to do for each artist I like!

See here for an amusing demonstration of the lack of randomness in computers

That link doesn't work (trailing quote). Try this one.

Yeah, you're right; the empeg guys should cryptographically hash the time and use _that_ as the seed. I would say "Hah! Proof that the empeg random shuffle isn't as random as it should be!", but nowhere has anyone stated that they _don't_ hash the time. Since the empeg has more than it's fair share of 'little details that make all the difference' I wouldn't like to bet against it!