jEmplode 42 pre 5

http://www.jempeg.org/jemplode20-stonecutters.jar

What's New:
1) Added CTIME tag support

2) Fixed copying from search soups that talk about "refs" (pasting would increment the refs and remove the node from the selection causing all the others to be off by one)

3) Transient soups colorize

4) fixed weird problem with refs soup w/ importing new tunes (just imported tunes would get stuck into refs = 0 soup for an instant, but just long enough to get synced). It would get cleaned up the next time automagically, but it turned the refs = 0 soup red. Kind of annoying. It's fixed.

5) Database wide deduping on import. This defaults to on, but you can turn it off in options. Basically, I compute a CRC32 on the data (i.e. non-tag) portion of the tune and append the length of that section to the CRC and use that as the hash for the tune. This hash is stored in a new tag for imported tunes called "hash". When a database is downloaded, i keep a Hashtable of Hash=>Node mappings so I can lookup a node at import time. This uses up more RAM (maybe 30 or so bytes per hashed tune) at jEmplode runtime and you take a performance hit at import time to compute the hash. I tried MD5 originally but WOW was it slow. I'm very curious to hear how this works for people and if they get false positives for duplicates.

6) Because of #5, mcomb's wish to have soup imports dedupe comes true.

Known Issues:
Still not checking for loops when you paste a playlist. For some reason I can't bring myself to actually focus and write this code ... Oh well.

I want to get the loop check in, and hear back from people about problems, then I'm calling this 42 and will likely also do an official release.

Mike

... and I also updated the version number ... that's a new feature for this one

Basically, I compute a CRC32 on the data ... I tried MD5 originally but WOW was it slow.

You might want to try Adler32. It should be faster than the CRC and it's built into Java (at least 1.3 and 1.4).

Ok here's the soup I am trying to do if someone could let me know if I can do it and how it would be great because I can't figure it out.

Artists
......0-E
........Albums
.............Tracks
......F-J
........Albums
..............Tracks
......K-O
.........Albums
..............Tracks
......P-T
.........Albums
..............Tracks
......U-Z
.........Albums
..............Tracks


Thanks again Mike for making all this cool stuff

and you take a performance hit at import time to compute the hash.

And it's very noticeable. I have all my tunes in a distributed filesystem, and I figured... eh, sucking them across a wireless network and then cramming them back is sucking like usual. So I switched to wired, and it was still slower than before (usually I remember to just do it wired right along).

Given that I usually batch imports and then go to sleep, I'll probably disable it, but it's a neat idea.

Layer 1: Tag Range on Artist 0-E,F-J,K-O,P-T,U-Z
Layer 2: Tag By Artist (If you want the actual name of the artist as a layer -- leave this out if you just want to segment by Artist first letter)
Layer 3: Tag By Album

Mike

This is the first shot at it, though ... From here I just need to find a better hashing algorithm. The only annoying part about that is that it means that it won't find tunes hashed with the old one. We'll see how it plays out

Oh cool -- I'll check it out ...

[moments later ]
on the same 16M file, I get the following results:

CRC32 = 1211ms
MD5 = 2644ms
ADLER32 = 3826ms

I'm going to look more into it when I get home though.

Ouch. The whole point of Adler32 is that it's supposed to be faster. Maybe someone has a better implementation than the one Java provides.

Edit: I realize that it's not Java, but these benchmarks show Adler32 running 2-5 times as fast as CRC32, which itself is 1.5-4 times as fast as MD5.

I suppose so much of it is implementation ... I'll look for an optimized implementation. The CRC32 impl I'm using is the one I ported from emptool. Not sure how it stacks up implementation wise, but it seems pretty quick ...

Just out of curiousity I benchmarked JDK 1.4's Adler32 against JDK 1.4's CRC32.. I ran it 3 times on the same 5 meg random dataset. Very strange ... Sun's Adler32 implementation must suck. Numbers are in millis.

java.util.zip.CRC32: 581
java.util.zip.Adler32: 1001

java.util.zip.CRC32: 591
java.util.zip.Adler32: 1002

java.util.zip.CRC32: 630
java.util.zip.Adler32: 1002

6) Because of #5, mcomb's wish to have soup imports dedupe comes true.

Sweet. Thanks Mike. For the speed issue, do you really need to compute a checksum for the entire file? Wouldn't a checksum for the first 10 or 20k be just as likely to produce a unique identifier?

-Mike

I thought about this one ... I may just do that. I would imagine after a certain amount of data, if it's not already unique, it's not gonna be. I'll try 20k and see how it goes. I probably need to have a UI for dealing with collisions too, but since i combine the length of the data portion with the CRC, I am hoping collisions will be very unlikely.

I'll try this tonight...

Or maybe the first 10k plus the last 10k?

Then again I dunno. In theory, a re-rip might have the same data at the beginning and end if you were trying to correct an error in the middle... And the whole point of doing the CRC checking is so that it correctly recognizes things like re-rips of the same tune and uploads them... right?

Well -- this is an interesting question. Some people would say that a rerip is the same song, some would say it's not. Odds of both the CRC _and_ the data-portion length being identical in two different rips is probably pretty slim though, so it would currently upload again.

Some people would say that a rerip is the same song, some would say it's not.

I don't see what the point of CRC checking would be, other than to make sure to detect re-rips on your PC and replace the tunes if they had really changed.

The benefit of the CRC checking from my perspective was that you can just drop any tunes in the soup and not have to worry if you've already uploaded them before.

To be able to identify a rerip, you really need a hash function that is designed for music (like a soundprint). Right now we're looking strictly at file contents.

So say you drop a tune on and the hashes match (i.e. you already have this tune on your Empeg). Right now it skips that file as a duplicate -- are you saying you want the old tune to be removed and the new one to take it's place instead?

Right now it skips that file as a duplicate -- are you saying you want the old tune to be removed and the new one to take it's place instead?

Exactly. Why else would I have a different version of the same song on my PC unless I was fixing a bad rip?

For tune-duplicate-checking, the tag information combined with file size is plenty. But if the actual bits are different, then you know you re-ripped the tune.

When you talk about doing soundprint signatures, that would identify the song, but not whether it was a re-rip. Two different rips would appear (correctly) to be the same to a soundprint signature.

And what about not just re-ripping, but upgrading the bit rate too? Say I've got an album at 128 on my player and on my PC. Then I re-rip at 256 and copy over those files on the PC, then drop them onto the soup. It should say, "ah, cool, this is a new rip" and replace all the FIDs.

The only way to have different data bits but be able to identify duplicates is a soundprint, though. Unless there's a cool open source sound print algorithm, I don't know that I'll ever be able to have two rips that i can identify as the same tune.

I think you need both checks, actually. If the strict-data hash matches, then you are just dropping the exact same song in, and that _should_ be skipped ("Oh -- i see you already have this exact song on your Empeg, I won't put it on a second time"). If not, then you would soundprint them and if the sound print matches then you know it is a different version of the same song ("Oh -- this is the same song, but the first match failed so I know the bytes are different -- I need to replace the tune").

Anyone know of a soundprint algorithm?

For the basic purpose you originally stated... Dropping a folder onto the player and not having it add the old tunes twice... wouldn't the file size and tag data be enough of a check to ID it as the same song?

Hmm, but then that wouldn't cover the bitrate-upgrade situation... And it wouldn't cover poor tagging practices... Hmm the soundprint stuff is looking better all the time, isn't it...

Didn't someone here on the BBS actually work on a soundprint application that went opensource, and we did a whole thread on it?

I thought so too .. I thought it was actually called "soundprint".

By the way -- the reason I can't trust tags is that you can change the tags outside of jEmplode but it's still the same song. So I have to create a hash of only the music portion of the file and store that with the tune on the Empeg. That way when you drop a tune again, I can hash the tune you just dropped and look for a match. But you're right -- the bitrate change would break that.

I'll go hunting for a soundprint...

No, it was called something else, but I don't remember the name. Arg.

http://sourceforge.net/projects/freetantrum/

Yup, that was it.

For tune-duplicate-checking, the tag information combined with file size is plenty.

I upload files I get from someone. A few days later I upload the same files with tags that are fixed with better/additional/correct data. They're still the same songs. I probably want an "option" to replace, but that might be too much work. I think what I don't want is both "songs".

Yeah -- There are really four scenarios here:

1) the tags are the same, the song is the same -- always should be rejected as a duplicate
2) the tags are different, the song is the same -- maybe we should reimport just the tags?
3) the song is different (but sounds the same) -- replace the song on the empeg
4) the song is different (and sounds different) -- add a new song

It looks like the code for songprint is only like 150 lines to compute the signature, but it uses FFTW (the Fast Fourier Transform library), which I'm sure is rather large. I need to either port FFTW to Java, find a replacement, or use the java wrappers and require the dll or .so (which would suck).

ms

The only issue here is that I'm just CRCing the file and it slows things down ... How long is it going to take to compute a soundprint I wonder?

maybe a check-box that asks lets you decide how you want this situation handled. skip, replace, or ask with each instance.

Sorry if this has already been offered...I haven't read through all posts.

back in your cage .. nobody told you to think

maybe a check-box that asks lets you decide how you want this situation handled. skip, replace, or ask with each instance.

Good idea IMO.

hey...being that tomorrow is my last day of employment...I wasn't thinking...auto-pilot.

There is a reasonably fast (and self-contained) CRC32 function that is part of the Hijack patch. Just grab any recent Hijack .patch file and scan through it for "crc32". There will be a table, and a function.

Cheers

If I understand the posts correctly, the main issue is the time it takes to compute the checksums that tell us if the files are the same.

Would a stepwise method improve the time?:
- check if the filesize is different. if different, we know we need to reload the file
- if the filesize is the same, check the tags. if the tags are different, then we need to reload the file.
- if the tags are the same compute the CRC. if the CRC is different, then we need to reload the file.

if the normal operational scenario is that most of the new files one adds are NOT duplicates, then this might speed things up (you only have to compute CRCs on files that fail the filesize and tag tests).

-ted

Most implementations of CRC32 are table driven, at least to some extent, and so are fairly fast -- once you step into the realm of real cryptographic strength hashes like MD5 and SHA-1, you're talking real computation. I am not familiar with Adler32, but if it is as simple as CRC32, yeah, you'd think it should much faster.

rjf&

I'm seeing occasional hangs when I "paste" a playlist I just cut. For instance, I have a structure which looks like (in terms of playlists)

shadow
/AliceCooper
/GreatestHits
/Alice Cooper
/HeyStoopid
/PrinceOfDarkness
/AllmanBrothers
/BadCompany

etc. If I cut "Alice Cooper" and paste it into "AliceCooper" then cut "HeyStoopid" when I try to paste it into "AliceCooper" the dialog box I right-clicked to bring up and choose paste from never goes away.

Likewise, if I leave "Alice Cooper" where it is and cut "HeyStoopid", then paste it into "AliceCooper" that works, but when I cut "PrinceOfDarkness", when I go to paste it, the paste hangs.

The same pattern seems to hold in the rest of the playlists.

The problem is that you have to be able to find the "matching" tune to compare against, so you have to compute the hashes up front to be able to lookup the hash to know what you're comparing against.... I suppose we could have several hashes (one on file size, one CRC, one soundprint) and each would narrow it down a lot -- I'll have to look at what the memory overhead would be to store those, though.

Probably the biggest issue right now is getting a good soundprinting implementation working in jEmplode...

Are any of these soup playlists or just regular ones? What OS are you on? I tried to create a similar situation on mine and I can't it to happen (on Win2k)... Do you cut with Right-Click=>Cut or Ctrl-X? Do you select from the tree or from the table? Same for paste -- do you right-click=>Paste into the tree or the table?

Also -- how are you launching jemplode? Are you just replacing the jemplode20.jar in your install dir and running the exe or do you have your own JDK and you're doubleclicking jemploe20.jar?

Yeah -- the CRC32 impl I'm using is table driven, and actually pretty fast... It only takes a little over a second to compute on a 16M mp3 over my network.

Right -- and at the risk of stating the obvious, the probability of collision (two byte streams generating the same hash) for CRC32 is much higher than for MD5 or SHA -- it's still pretty small statistically speaking.

rjf&

I tack on the length of the data portion of the file to the CRC value, so the odds go way down. CRC32 collision rate is somelike like 1 in every 2^32, and when combined with the input size length, it's really low......... I hope.....

That was it. I didn't know I could put all the ranges on one line and was making a line for each which didn't work very good.

Thanks

Don't you love programs with no documentation that little label about the text field was my attempt at docs .. oh well .. i don't have a future in technical writing.

Are any of these soup playlists or just regular ones?
All regular

What OS are you on?
Linux (RedHat 7.2)
I tried to create a similar situation on mine and I can't it to happen (on Win2k)...
I can't make it happen consistently either. In fact, after changing those playlists by making one cut/paste at a time and syncing in between I haven't gotten it to happen again yet since.

Do you cut with Right-Click=>Cut or Ctrl-X?

Right-click.
Do you select from the tree or from the table? Same for paste -- do you right-click=>Paste into the tree or the table?

I was doing it all in the tree.

Also -- how are you launching jemplode? Are you just replacing the jemplode20.jar in your install dir and running the exe or do you have your own JDK and you're doubleclicking jemploe20.jar?

I replaced the jar and run the executable.

I'm trying some things to see if I can get it down to something that seems reproducible, and so far, losing.

Do you get any errors on the console? Also, try using Ctrl-X/C/V and see if it goes away .. Curious if it's hanging because of a weird UI thing or because of a more sinister problem.

Even with no documentation I have gotten better support with jemplode than and piece of software I have purchased.

Do you get any errors on the console?
Which console?

Also, try using Ctrl-X/C/V and see if it goes away

I have to break it again before I can see how to unbreak it.

I made a soup like this in jemplode only and it doesn't add all the tracks that are in the albums even though the tracks have the same source.

Any ideas ?

Can you send:
1) the value of the "soup" tag of the soup playlist (from properties=>advanced)
2) a screenshot or something of the table view of the album
3) a screenshot or something of the advanced tag properties of one of the tunes that's not making it in?

I setup this soup on mine and it _appears_ to be working properly for me, but it's so hard to tell...

I take it back .. That's too annoying. Go get

http://www.jempeg.org/debug.zip

unzip jemplode20.jar, unzip debug.zip on top of it (it should replace about 6 or 7 files) then zip jemplode20.jar back up again. Alternatively you can unzip debug.zip and then zip -ru jemplode20.jar to update it and replace the files. Then run jemplode again and send me the console output. If you don't get console output, open jEmplode 2.0.lax and search for "stdout" and you should see a line that defines where stdout redirects to .. set that equal to "console" or a filename and then rerun.

What this does is print out the name of each soup, the total number of tunes in the soup vs. the total number of tunes on your Empeg and if there are any in the db that aren't in the soup, it prints out which ones. For a soup that doesn't have any searches in it, the total number of tunes in the soup should always equal the total number of tunes on your Empeg. If you have a search anywhere in the chain, the search can filter out some, so it's ok to have missing tunes in that case.

You may want to keep a copy of the original jemplode20.jar, since this debugging slows down soups a lot.

instead of just a crc32 check of 10k in the beginning and 10k at the end, why not borrow from gordian knot compression check(divx related) and do a check on X#bytes every Interval. so say, 1000bytes every meg of the mp3.

this will certainly eliminate the possibility that a song can be the same in the begining and end, but not the middle because of a rerip.

That's not a bad idea ... It will be interesting to do a performance comparison between full MP3, first 20, first 20 + last 20, 1k every X bytes.

go into jEmplode 2.0.lax ( I think that's what it's called) and search for "stdout" .. There should be a line that tells where stdout redirects to ... You can set that to "console" or set it to a filename to log. Then when you get it to happen again, see if there's an error in there.

Done. So far I got one more hang using the right-clicking and no more with ctrl-v to paste. I'm still trying.

Very interesting ... So it must be some race condition with the UI thread.

Wow .. performance goes WAY up this way... I took 1000 byte block every 50000 byte interval (roughly 100 samples on a normal MP3) and it dropped from 8 seconds to 300 ms. I'd love to know, mathematically, how this affects the odds of a collision.

Still haven't failed it using just ctrl-v to paste and using right-click to cut.

Did find another problem which maybe you can reproduce.

Except that I didn't. Text of not a problem removed.


I see it's that I was playing with 10000 items, but I just blew up JEmplode entirely but cutting a playlist which was listed twice in the same parent playlist and then trying to paste into same.

The Java VM seems to want to take responsibility.

Post deleted by dbrashear

Actually, it's not a hang, it's just a very, very long update. I had left the first one I "hung" around, and some time later, it finished updating.

I guess the answer is either "get a faster machine" or "don't play with 10000 items at once"

Oh .. you know what's happening? When you have a table open, and you change tags, the table receives update events which causes the tree to resort. Every single thing you update is causing the table to resort (i.e. a sort per change). This is on my list of optimizations to work on... Definitely having 10,000 items at once is making it worse, but it is jEmplode's fault that it's happening ...

Sounds like you need some kind of freeze/thaw mechanism when you know you'll be firing a lot of events. Then if events arrive while the UI is frozen, you don't act on them until it's thawed (and you act only once). Is that the right way to go?

I think you'll need to do some action per-event, but things like sorting the items can wait, right?

You shouldn't need to do much sorting anyway, if you can use something like the Arrays.binarySearch() method when doing inserts and keep things permanently sorted. Then only user-requested sorts need action...

In reply to:

---

You shouldn't need to do much sorting anyway, if you can use something like the Arrays.binarySearch() method when doing inserts and keep things permanently sorted. Then only user-requested sorts need action...

---

That's actually exactly the optimization a was thinking of. Right now there are some problems with what parts of the system know what pieces of information (for instance, if you change a tag that isn't a tag that is being sorted on, the playlist doesn't need to do anything -- things like that) that I need to work out. It's kind of been sitting in m Eclipse task list for a while now and I can't remember why I didn't do it originally (there was something annoying about the problem).

I figured it was something like that. At least it's not a bug, really; It's easy to deal with when you know to expect it.

And I've still been using ctrl-v to paste with no problems there either.

So did the right-click thing turn out to not be hung too? just a huge update?

In reply to:

---

Wow .. performance goes WAY up this way... I took 1000 byte block every 50000 byte interval (roughly 100 samples on a normal MP3) and it dropped from 8 seconds to 300 ms. I'd love to know, mathematically, how this affects the odds of a collision.

---

Well, its been a while since i've gone to a statistics and probabilty class, but here's my best guess:

1/(2^32) is the chance for a collision on a regular crc32 check.
you would multiply this with the probabiliy of a 50000 byte data stream that is the same the 1st 1000 bytes and different the rest.
then each segment lowers the proability of a collision. so if:
x byte block every y byte interval, with z samples, then the probability of a collision is:

[ 1/(2^32) ] * [ (yz)/(y-x)) ]

in other words, for your example, [(50000*100)/(49000)] * [1/(2^32)]

hahah. hope that makes sense to you cuz it surely doesnt to me =)

This may be true of random data but I am not sure it is true of music and perhaps even less so of an mp3 file. I wonder if you change 1 bit in a wave file how many bits change in the resulting mp3 file? In a video mpg file because of the index and difference frames a small change would propagate through many following encoded frames. This should make it more likely that intermitant crcs would catch a change.

However, it still seems likely that a rerip to correct a single skip in a song could easily be missed. Somone should probably run a test.

On the other hand I would be perfectly happy to flag my ID3 files on rerips if we could create a standard for that??

Cheers,
Eric

I take it back .. That's too annoying. Go get

http://www.jempeg.org/debug.zip

unzip jemplode20.jar, unzip debug.zip on top of it (it should replace about 6 or 7 files) then zip jemplode20.jar back up again. Alternatively you can unzip debug.zip and then zip -ru jemplode20.jar to update it and replace the files. Then run jemplode again and send me the console output. If you don't get console output, open jEmplode 2.0.lax and search for "stdout" and you should see a line that defines where stdout redirects to .. set that equal to "console" or a filename and then rerun.

What this does is print out the name of each soup, the total number of tunes in the soup vs. the total number of tunes on your Empeg and if there are any in the db that aren't in the soup, it prints out which ones. For a soup that doesn't have any searches in it, the total number of tunes in the soup should always equal the total number of tunes on your Empeg. If you have a search anywhere in the chain, the search can filter out some, so it's ok to have missing tunes in that case.

You may want to keep a copy of the original jemplode20.jar, since this debugging slows down soups a lot.



I think I did everything right I set stdout to redirect to a file but I don't get anything and I am not sure what you mean by console. What should I be watching.

Thanks

So did the right-click thing turn out to not be hung too? just a huge update?

No, the last time it happened I was in the tree, which isn't very big, and I let it sit for rather longer.

Ok I think it works now (the soup view) I think I was leaving it set on track when making the range view instead or changing it to artist. DUH sorry for the user error.

Thanks

I'm porting the songprint algorithm to Java ... It converts MP3 to a PCM bitstream, so combining songprint with file hashing should allow us to identify all the scenarios we need. I have no idea what the performace of songprint is like, though.

In reply to:

---

However, it still seems likely that a rerip to correct a single skip in a song could easily be missed. Somone should probably run a test.

---

i totally disagree. the skip that is encoded will certainly change the mp3 at some point. it will change the sequence of frames, and will alter the bitstream from that point.

what X every Y interval will do is catch a song that has been mp3trimmed. if there was a corrupt frame that was taken out, 10k from begin/end will not see it.

I'm porting the songprint algorithm to Java

Cool. Aside from the obvious advantage of duplicate handling it seems like that may have some other uses as well. Would automatic id3 tagging of files be a possible future use (that was the original reason for the songprint stuff right)? Or how about automatic detection of damaged files if the songprint does not match what the id3 tags say the file should be.

-Mike

Yep -- I don't know that there's a songprint server anymore (freetantrum.org seems to be gone). But perhaps someone could set one up for Empeg folks to submit their data to. I've ported the songprint algorithm itself, but I need 1) a good FFT library for it and 2) an mp3 decoder (basically, I need a PCM stream from an MP3). For 1), one option is the FFTW Java wrapper (which uses JNI), and for 2) I can use JavaLayer (pure Java). There is obviously a big issue of performance when computing this stuff, and I don't know if Java's going to be up to it.... We'll see, I suppose.

ms

This should be a simple one:

How about a little advanced search cheat sheet somewhere, so I don't always have to refer to the FAQ to remember what I can search on?

Sigh.

Okay. This was my first attempt at uploading with these new PrePre jEmplodes and it didn't go well at all.

The main problem is that it failed to upload everything I told it to, and it also seems to have failed rebuilding the players database, as after the ``sync'', the player took quite a while on that item in the startup phase.

To be complete, I first dragged a folder from Windows Explorer to the main playlists item. Oddly, it only inserted the subdirectories of that folder (that is, I dragged an artist folder, and it put album playlists at the top level and the artist playlist was nowhere to be seen). So I created the artist playlist and cut the (single) album playlist, clicked on the artist playlist and clicked paste (all with the toolbar buttons).

The next folder I dragged into the opened main playlist window and it seemed to insert properly.

I then placed the two new top-level playlists in the correct position. (BTW, is there any faster way to do that than clicking the up/down buttons multiple times, like emplode's Alt-drag?)

Then I synced. I wasn't paying close attention, but everything seemed to go fine until towards the end, when it seemed to stop before it uploaded all of the tracks. It also took a long time to restart the player, as it was rebuilding the database(s). I noticed that some of the tracks were still red, so I tried to sync again, but jEmplode didn't try to upload anything, but it did reboot the player again, taking a long time. Then I exited jEmplode and it told me that I should sync. I told it not to bother, since I was sure it wouldn't do anything.

I started up emplode. None of the new playlists were there. A ``refs=0'' search showed many, but not all, of the tracks I'd intended to upload. I deleted them and followed the same procedure again, except under emplode, and everything worked fine.

This is pre5 under Windows 2000 Pro. I'm not sure what JRE jEmplode is using. I'm double-clicking on the jEmplode.exe executable, but I have both 1.4.0_01 and 1.3.1_04 installed. If I double-click on the jar file, the toolbar buttons are wider than when I run the exe. This bug was under the exe. (Maybe you could put the Java version in the about box?)

The player is running beta13 with Hijack v291.

Edit: This is all via ethernet. Is it possible that the use-hijack option might be screwing things up? ISTR that that's the default, and I don't believe I've touched that option.

I think freetantrum is dead, looking at sourceforge and other supporting sites, last posts were 2001.

But there is Musicbrainz which does appear up and running, and has a Linux distribution.

Worth a look see?

I'll add a "Add Variable" button in the advanced search screen

I want a soup playlist like msaeger e.g.

Artists
......0-E
........Albums
.............Tracks
......F-J
........Albums
..............Tracks
......K-O
.........Albums
..............Tracks
......P-T
.........Albums
..............Tracks
......U-Z
.........Albums
..............Tracks

But what I also want is to exclude an artist from the tree if their album track count is (say) < 4.

I want to do this so that the soup more or less just lists full albums by artist and ignores single track entries from Various Artists albums.

Is this possible?

Currently this is not possible..

Hijack is used for downloading but not uploading .. However, it is possible that Fast Connections are screwing you up.

1) when you dragged a folder, the folder itself wasn't added. this is my first concern because it's weird -- I just tried this on WinXP and it works (i.e. I drag folder "X" with folders in it, and I get an "X" playlist with all that stuff in it). Can you try to duplicate this and tell me if an exception shows up on your console? was there already a playlist at the top level that had a name that was the same as the folder you dragged?

2) Right now I don't have drag-n-drop hooked up from jEmplode to jEmplode... I'll put it on the list.

3) did you get any exceptions on the console when the sync failed? It's weird that it never displayed an error ...

4) As far as the playlists not being there -- The deal is that because you are now allowed to cancel an upload, jEmplode tries to not leave a bunch of zero ref playlists on the player (i.e. it uploaded a playlist but it didn't upload the playlist that actually referenced it). So it puts all the playlists at the end of the sync. The consequence of this is that if sync fails during one of the tunes, none of your playlists get uploaded.

I'm not even sure where to start with this one I'm just going to go and try to do some big uploads and see what happens.

I don't even know where the console would be. Fill me in and I'll try again. Note that that report was about the one and only time I attempted this with pre5 (or any of the prepre releases), so it's possible that it was a freak one-time occurence. Tell me where to find the console and I'll try to reproduce it. But if the console is supposed to be a pop-up window that shows the exception, as I've seen before, I definitely didn't see one of those.

1) I realized I wasn't totally clear about exactly what I did. I think you got it, but let me reiterate. When the top-level playlist wasn't added, I dragged a folder from explorer onto the left panel's root playlist item (BTW, it didn't highlight -- should it? If not, can you add that?). The second time, when it worked correctly, I dragged it into the right panel while the root playlist was open. Unfortunately, there was another difference. The first one was a folder that contained a folder that contained some mp3s. The second was a folder that contained tracks, without an intervening folder. I'll try to narrow down exactly what caused this problem later (when you tell me where to find the console). The only other thing I can think of that is not normal is that I'm dragging these folders from Explorer's location bar (where the URL would be if I were displaying a web page), and not from the main pane. I've never seen that cause problems with any other application before, but it's not 100% normal.

The only other things I can mention are that it wasn't exactly a large upload (two albums worth -- about 110 MB) and that both of the folders contained m3u files. One created a sub-playlist that needed to be deleted before the sync (I made sure not to delete any tracks during that process) and the other one was bogus, referencing files no longer there, which was reported when it was added.

I can't seem to duplicate that failed first-level playlist thing. I suppose it's possible that I just brain-farted on that one and dragged the wrong thing.

Oh, and I just started jEmplode via the command prompt with ``java -jar jemplode20.jar'', so I've got a console now.

Well, I can't seem to duplicate any of those errors now.

But I'll try some more later on tonight. I think it might have to do with that bogus m3u file.

I'm trying to get the download feature working properly in jEmplode v42-pre5 and am having some problem...

* The tags aren't filling in properly...

I have "Write ID3v2 Tags to Downloaded Files" selected, but the genre and track numbers aren't filling out... I'd LOVE to have the track number use the position fields for population as a lot of my tracks don't have a track number properly populated...

* I'm using "{artist} - {source} - {pos:2} - {title}.mp3" as my filename format, but the POS has mixed results. Sometimes I get no POS at all, sometimes I get a single digit...

Any help?

- Jon

Small feature request:

Could we have an option to ignore playlist files on import?

Great stuff Mike..I've just gotten around to upgrading

I think I've found a bug:
I created a Soup on the Title tag, and it only listed about 1/3 of my tracks. When I created a Ranged Soup on the Title tag, all tracks appeared correctly in their respective ranges. (Both done 'JEmplode only')

Thanks for adding Wendy to the mix, can I now request that I can (optionally) recusively apply Wendy flags to playlists (including Soup).

BTW, If I rename a Soup (on empeg) playlist will it stop syncing itself? I really want to rename my Soup lists to always appear at the end, and also make them more...aesthetically pleasing... eg "(By Decade)", "(By Artist)", "(By Genre)", but I don't want them to lost their autosync status.

Something I noticed that could be better (though doesn't do any serious harm):

I created a soup based on play_count, so that I could see which tunes I overplay (and to stress-test your code ). I have set play_count to have java.lang.Integer as its typeClass and sortClass, like so:

jempeg.tag.play_count.sortClass=java.lang.Integer
jempeg.tag.play_count.typeClass=java.lang.Integer

But the soup view isn't sorted in numerical order, as the attached image shows. How difficult would it be to make the soup views sort using the same class as the corresponding column? I'll give you a hint: the answer we want to hear is "not very"

I officially seem unable to reproduce any of those errors I reported before. So nevermind.

you mean ignore .m3u files?

It's supposed to, but then I haven't taken a look at the tag definition code since before the 42 releases, so maybe those system properties just aren't being loaded in properly.

<mrburns>eggggsellent</mrburns>

All of your tracks should be in there, but I'll take a look ... Souping on title is kind of a weird thing since you will very rarely have duplicate titles (probably).

Soups should continue to sync if you rename them (the soupiness is saved in another tag called "soup").

I'll look into the Wendy request.

ms

That's what I mean, as well as .pls files and whatever other sorts of playlist files there might happen to be. This option exists in emplode proper.

The reason is that I have .m3u files in my album folders, which makes it easier to play an album. But that means that on jEmplode directory imports, it creates the album and then a duplicate as a child of that album playlist. Which is annoying.

Plus, I have the feeling that some bogus m3u files might be what caused those errors I saw earlier, which I'm still trying to reproduce, despite my formal rescindment.

mike,

i've been toying with the soups for a few weeks now, and i got a few soupy questions.

#1: i originally created the soups before the "sort by" option was added. when i updated jemplode to a new version, i opted for a "sort" by "tracknumber". i expected all of the soups to be resorted, but they have not. do i need to delete, and then recreate the soup?

#2: if i change sort/tag criteria in the "soup tag" , when do the changes take place? i've changed the search criteria on my soup, and didnt see the soup update. (these were soups on the rio.)

#3: is the sorting only available by one field? (i.e. can i sort by album, then by tracknumber?)

#4 a request: do you think you will be able to implement a "sync" feature that can maintain my computer and rio in sync? i know this feature has been asked for alot in the other forums.

thanks man!
-ted

#1 right now the sort by feature doesn't change the sort of soups ... there are some tricky problems with the way soups work that assume that they are sorted by title ... it's on the list of things-to-do

#2 i don't do much testing with changing the search criteria (as opposed to deleting and recreating). However, what should happen is that you will have to sync to send the new soup setting down to your Empeg, then when it redownloads after the sync, I would _think_ that is when it would be adjusted. Then you'd have to sync again to get the newly added/removed tunes down.

#3 right now you can only sort by one field at a time... this has been requested too, but it's not a really high priority right now... I'll make sure it's on the list though

#4 this has definitely been talked about ... there are a couple ways to do it, and it probably just needs some discussion to decide the best way. I think the best way, albeit the slowest way, is to compare all the hash values of your tunes on your PC and your Empeg and kill or sync ones that don't match. I think you would have to designate one of the locations as the "master" for this process, so you know whether a tune is newly added on one side or freshly deleted on the other.

An alternative way would be to store the original full path to the MP3 in an Empeg tag. The problem here is that while it is very fast to then do the lookup, it breaks terribly if you move your MP3's around at all (i.e. you decide to do artist directories on your PC or something).

Any other ideas for syncing would be appreciated.... How does rsync do it?

Am I the only one experiencing this stuff:

* The tags aren't filling in properly...

I have "Write ID3v2 Tags to Downloaded Files" selected, but the genre and track numbers aren't filling out... I'd LOVE to have the track number use the position fields for population as a lot of my tracks don't have a track number properly populated...

* I'm using "{artist} - {source} - {pos:2} - {title}.mp3" as my filename format, but the POS has mixed results. Sometimes I get no POS at all, sometimes I get a single digit...

I posted this a while ago, but no one said boo...

- Jon

I'll put these on the list...

Heres a presentation on rsync that explains it pretty well. It splits up files and does a MD4 checksum on each piece, then it bundles those up into the signature.

http://olstrans.sourceforge.net/release/OLS2000-rsync/OLS2000-rsync.html

Sean

I just browsed through the RSYNC paper, and if I understand it correctly, it does more than identify if a file has been changed... it tries to locate the change happen, so only the delta needs to be sent (like the unix "diff").

Wouldnt that be overkill for what we want here? we just want to ID if the file has changed, and if so, just overwrite the RIO file. That should be an easier checksum. Perhaps, even the same algorithm that is used now for checking for dups?

-Ted

I think you're right ... Though that rsync paper was really cool to read through In our case, the files are rarely being _modified_, rather they're mostly being added or removed. So we really only need the full-file checksum that we do now. The difference in our case that makes things tricky though is that we lose filename + directory structure on the Empeg side, so we have to _find_ the files on the PC side. One way around this is to keep an index file of computed hash codes at import time, but that's really just as bad as storing the source filename in a tag, since if the file moves or changes name, they'll no longer be in sync. It's just a tricky problem in general.

I tried pre5 and noticed two problems that nobody seems to have mentioned yet.

I can add a folder of songs for uploading just fine. But I am unable to add any single tune for upload.

The other problem involved a failed sync. Everything went a bit funny on me, views were not refreshing and such. So I 'reconnected' to the player and everything came up just fine. The batch of songs that was being uploaded never made it to the player (not even found with refs == 0) but I couldn't re-add them until I disabled duplicate detection.

Perhaps that is not too much to ask for: if you change your directory structure, then the files would have to be recopied.

-t

Any chance we could get upload support for FLAC and Ogg files?

Greg

It's not a very high priority ... since the player can't play them (natively), there's probably not a whole lot of demand right now. I'd need to port a parser for the tags/headers for both of the formats to properly import them, too, which would probably take a little while. However, you should be able to turn on "allow unknown formats" in options and just upload them as non-tune files.

FLAC files just use id3 tags like mp3's so as far as parsing goes, it shouldn't be a real problem. The main thing would be to see the .flac extension and set the codec to "flac". Ogg, on the otherhand, does their own thing on tags for whatever reason. In another project, I just use the file name as the title and artist and set the codec to "ogg".

The reason why I ask all this is just to get the ball rolling. I don't think it'll be long at all before RioPlay starts getting some use, ogg and flac are probably the star attractions to it. Having JEmplode support these at least on a rudimentary level would go a long ways.

Greg

In reply to:

---

FLAC files just use id3 tags like mp3's so as far as parsing goes, it shouldn't be a real problem.

---

Small correction... FLAC uses Vorbis comments now. libFLAC has an interface for getting at them. The format itself is neutral towards id3 tags. The FLAC plugins now read all three (vorbis, id3v1, id3v2) but it's up to each individual app to support them.

Josh

As readers of General know, I'm currently uploading the Harry Potter audio-books.

I tried using jemplode to put the tracks on the player, with some success, but found that the metadata were wrong for these monaural tracks. The length is marked as twice the actual length, and the bitrate is recorded as "vs0" instead of "vm48". I tried with emptool and it also got the length wrong in the same way, and the bitrate wrong in a slightly different way.

The jEmplode upload is the highlit one (blue background) and the emptool one is white background:



For comparison, local tools report:

Title: The Boy Who Lived
Media Type: MPEG 2.0 Layer III
Audio: 49.766888 KB/s, 22KHz (mono)
Length: 10:37

[Edit: it's possible that it's confused by being MPEG 2 Layer 3 and the lower sample rate, rather than by the number of channels]

Some more investigation - I encoded the same track in four different ways: mono 44kHz, stereo 44kHz, mono 22kHz, stereo 22kHz. All four appeared as vs0 in jEmplode, and the last two had the wrong duration.

So the duration bug seems to be related to the use of low sample rates and MPEG 2, and the rate bug appears to be universal.

The MP3 header parsing code is a straight port of emptool's ... I need to talk to dmz and see if his ID3 parser happens to also parse that data too -- maybe I can just use theirs instead.

The bug report I submitted against emptool is probably of use, as it contains a bit of extra research. Basically, emptool always thinks tunes are stereo, and gets the length wrong when it's a 22.05kHz sample-rate. (I didn't try other rates)

You may find some additional useful information about your problem in this thread.

tanstaafl.

Ah, I'd forgotten about that thread, despite being a significant contributor.

Note that the current problem is on the upload side, not the player (in fact, I took to testing this without going as far as synchronising in jEmplode).

But if the player didn't handle MPEG 2.0 until recently, then it's no surprise that the upload tools have problems too. Thanks for the pointer, Doug.