Do you feel lucky punk? jEmplode 46 stumbles in

OK -- So note that while this doesn't appear all that different outwardly, there are some pretty substantial changes behind the scenes. Not to mention this thing was in a billion pieces on the floor not too long ago. I lead in with this to point out that there will probably be some weird problems (the entire synchronization system has been rewritten to accomodate the Karma).

Basically consider this alpha quality for right now, but it would be nice to get some feedback. I'm not putting it on the real jEmplode site yet, though I probably will for the next release.

http://www.inzyme.com/rio/install.htm

ms

I'm already livin' on the edge... might as well download it and try to shoot myself in the foot. I should have it installed in a few seconds

Ask and you shall receive. Apparently my sync attempt trashed my database. When it finishes rebuilding, I'll try again.

Not likely a jEmplode problem: see next message

Looks like I had the empeg plugged into a bad power cable which cut out at just the right time...

Meaning jEmplode = not at fault, or is it still suspect?

I doubt jEmplode had anything to do with it. It was just unfortunate timing; I plugged the empeg into one of those cell phone power bricks whose cord was apparently damaged earlier to try it, updated some playlists and tried to sync, lost, and then while it was rebuilding the database on player restart I happened to nudge the serial cable slightly (I have it consoled for some debug work) and it rebooted. Fiddling with the cable shows it's no good. Oh well, at least it was a sleazy adapter and not one of the nice, small ones.

Downloading now. Should I be seeing how it works on the Central, too?

Wow ... Speaking of feeling lucky .... You can try ... No idea what it will do.

nice dog.

That would be Sydney ...

first bug. on a virgin empeg, it can't create a 1st database. heck, not even emplode 2.1 can do it. had to use v2final.

here's the relevant log when i tried to sync.


e2fsck 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
ext2fs_check_if_mount: No such file or directory while determining whether /dev/
hda4 is mounted.
! fidfile_disk.cpp : 968:Why did write return zero for offset=0, size=0?

woof, woof.

It uploaded one album with no problems. I like how you made it so I can browse the playlists while syncing.

I'm not brave enough to try it on the central

I'm not brave enough to try it on the central

Seeing as how the prior version of Jemplode that I tried on the Central tried to delete everything, that's probably a good idea.

Been busy, haven't tried this version yet...

How would I get my empeg back into this state to test this out?

Disk builder ?

first impressions.

1) the ability to interact with the interface while syncing is dope, but takes a lot of getting used to. theres no indication on the main screen when i hit sync that anythings happening. can the transfer bar share the bottom half of the playlists tab? the details can stay where they are.
2) whenever i sync with emplode, when i go back to jemplode, it gives me a "Playlists referred to a missing tune or playlist with ID 272".
3) dupechecking is excellent and hashes are calculated really fast!
4) windows shortcuts (.lnk files) arent implemented yet =(.
5) tags changed on the hdd doesnt change the tags on the player if it recognizes as a dupe.
6) some of my files are not syncing thru. it gives a UTF16-LE error for the reason.
7) a while back, we had a discussion whether "remove completely" will apply to fids with more than one ref. right now it deletes it completely and changes all playlists that contains it. can we get a toggle where it wont do that? especially useful when i have a favorites playlist that contain singles and refernces to albums i want to clear.
8) so where's the autosync to hdd feature of the karma?

as you can see, a lot of this is is what i was hoping would be in jemplode. sorry. but the #1 thing that is most important is the quick hash generation. too bad i have to upload all my files again to generate the new ones. yay to 60gigs of uploading this weekend!

Interesting. It seems to behave better with the Central than the prior version did.

I get one error message when connecting which says "'All Music' referred to a missing playlist with ID number 272", but then it correctly displays all artists and albums, something the prior version didn't do.

Note that "All Music" is displayed in Jemplode in the same place that the root playlist of the Empeg Car would be displayed. Only I don't think you get to choose the name for it on the Central like you do on the Empeg Car.

I'm not going to try Synching with the Central yet, since "All Music" is coming up red, but I just wanted to say things are looking promising.

I had about the same error with the empeg but after I did the sync it went away and "All Music" wasn't red anymore.

turns out my player was stil v2b11, and that was what was causing it to fail (and give a weird FSCK error in jemplode). upgrading to v2final allowed jemplode to build the initial database on the virgin empeg.

272 = Unattached Items ... So what it calls "All Music", in jEmplode + RioCar it calls "Playlists", right?

All Music is most likely coming up red because it added Unattached Items to it ... You should be able to open the Transfer Details tab and see exactly what changes are pending (nice side effect of having access to that queue now).

I lied a little bit ... 272 is Unattached Items, but it's not that jEmplode has added it, it's that jEmplode has REMOVED it. I took out the Unattached Items check inside of jEmplode so it wouldn't break the Karma. I guess I probably need to add it back in as an optional setting. I believe what happens is that you don't actually have an Unattached Items playlist on your player, rather it's kind of faked. So Emplode creates a phantom 272 playlist to fulfill the reference that the root playlist has for it. I seem to recall old jEmplode did that also (made a non-syncing addition of Unattached Items in the UI).

more stuff.

1) jemplode allows you to exit while a sync is happening. really bad, since the bar is invisible to an unknowing user. did that accidentally.
2) you try to add some files (about 60 albums), not sync and revert changes, then readd the files, and the transfer queue goes crazy, with a lot of duplicates and nullpointerexception. my guess is that the revert changes function doesnt clear the queue correctly. didnt even want to see what happens if i hit sync.
3) starting jemplode right pane is on nothing. if you drag files to the pane, it goes to the all soup without getting into any playlist. suggest to default starting the right pane in the "Playlists" folder.
4) i was able to crash the v2final player by doing the following. i tried to upload the 60 albums again, then before syncing, deleted everything with "remove completely". it left me with a lot of "Updating Playlists" in the transfer details, so i synced, then without restarting jemplode, i readded the albums, synced again. can recreate the crash doing the same steps.
5) using the cancel button in the sync doesnt work. it keeps on going. you need to highlight all the items and rightclick, then cancel for it to actually happen.
6) WMA files arent supported.

272 is Unattached Items, but it's not that jEmplode has added it, it's that jEmplode has REMOVED it.

Hence JEmplode's desire to delete everything from the Central when you press synch? Everything on the Central is an unattached item, playlists aren't used by default unless you specifically create them.

why does jemplode shoot a UDP packet on startup?

Protocol UDP Connected to 44.243.178.24 Port 45592

Are they actually in Unattached Items or are they just unattached (slightly different) ... In current Empegs, it's legit to not be in any playlists (including unattached items), but the Unattached Items playlist is still secretly there (Emplode hides it with some magic). I just fixed this, though, so Unattached should reappear in jEmplode (next version).

ms

Broadcast discovery ... It broadcasts a '?' on the empeg's port and any Empegs on the network respond to the request. If you don't need discovery, you can turn it off in the Select Connection Dialog's options menu

I'm telling you this like you didn't write the faqin FAQ.

Are they actually in Unattached Items or are they just unattached (slightly different) ...

I have no idea. All I remember is...

- Back in the old days of the car player, if something wasn't in a playlist, it was an unattached item, and that was a bad thing.

- When they added soup views to the car player and to the central, it became OK for there to be a song that wasn't in a playlist. So they hid the Unattached items and aren't displaying any errors about it any more.

I could be remembering all of that wrong, hence the reason I'm not understanding what you mean by there being a slight difference between Unattached and unattached.

I'm telling you this like you didn't write the faqin FAQ.

Heh, you know orders of magnitude more about this stuff than I do. I've looked at the synchronization code enough to know that it would be dangerous for me to try to grasp it.

Mine was missing ID 2, when it recovered. I guess I should try another sync. While it was rebuilding I went out for food, and had an adventure I will post details aabout to OffTopic shortly

In reply to:

---

can the transfer bar share the bottom half of the playlists tab? the details can stay where they are.

---

I agree that it's weird there's no indication at all .. RMML actually shows the currently syncing file name in the status bar ... I'll play around with this.

In reply to:

---

whenever i sync with emplode, when i go back to jemplode, it gives me a "Playlists referred to a missing tune or playlist with ID 272".

---

Fixed

In reply to:

---

windows shortcuts (.lnk files) arent implemented yet =(.

---

writing this now ... .lnk format is officially the most overengineered file format i've ever seen

In reply to:

---

tags changed on the hdd doesnt change the tags on the player if it recognizes as a dupe.

---

yeah .. this is a known limitation (of emplode and RMM too, I believe). this is, in general, a hard problem because you can change tags independently in each location, so there's no way to know (automatically) which direction they should sync. I suppose this could be a configuration option though, if you maintain your repositroy in a single location.

In reply to:

---

some of my files are not syncing thru. it gives a UTF16-LE error for the reason.

---

Are these all WMA files that do this? Is there a particular file that always fails? If so, can you send me a link to download that so I can try it here?

In reply to:

---

a while back, we had a discussion whether "remove completely" will apply to fids with more than one ref. right now it deletes it completely and changes all playlists that contains it. can we get a toggle where it wont do that? especially useful when i have a favorites playlist that contain singles and refernces to albums i want to clear.

---

You can get to "remove completely" a couple different ways -- what UI method are you referring to? I think I'm misunderstanding you request, though -- if you don't want remove completely to delete tunes in multiple places, wouldn't you just not say "remove completely" (and just do a regular remove)?

In reply to:

---

so where's the autosync to hdd feature of the karma?

---

the only autosyncing that RMML does is in the sense that when you start making changes, it will automatically begin to sync in the background ... It doesn't support any of the sync profiles or anything -- is that what you're asking about?

In reply to:

---

1) jemplode allows you to exit while a sync is happening. really bad, since the bar is invisible to an unknowing user. did that accidentally.

---

Hmm .. whoops .. that's supposed to ask you if you are sure (if there are pending transfers).

In reply to:

---

you try to add some files (about 60 albums), not sync and revert changes

---

I'll check this out ...

In reply to:

---

suggest to default starting the right pane in the "Playlists" folder.

---

shouldn't be a big deal -- i'll try it out ...

In reply to:

---

upload the 60 albums again, then before syncing, deleted everything with "remove completely"....

---

You get the Evil Tester Award ... I'll check this out also.

In reply to:

---

using the cancel button in the sync doesnt work. it keeps on going. you need to highlight all the items and rightclick, then cancel for it to actually happen.

---

Hmm .. OK.

In reply to:

---

WMA files arent supported.

---

It's supposed to be -- Though I don't actually HAVE any WMA's, so it doesn't get tested. You're not trying to upload DRM'd WMA's, right? That will definitely fail. What errors are you getting? Is this the UTF16-LE one?

attached is what i see. i've pm'd you a url to download the songs in question. be patient, i only have a 12Kb upstream.

You can get to "remove completely" a couple different ways -- what UI method are you referring to? I think I'm misunderstanding you request, though -- if you don't want remove completely to delete tunes in multiple places, wouldn't you just not say "remove completely" (and just do a regular remove)?

here's my dilema. say i have a lot of mixed cds with various artists a lot of the same tracks appear in these albums. to save room, i start using only one copy of the song on the empeg, and referring to it in the corresponding playlists. (on my windows box, i actually have all the songs in one big singles directory, and individual folders containing .lnk or .pls files.) so say "nsync-bye bye bye" (note, i dont have this, nop wisecracks pls) appears on 5 of the mixed albums. so with my current scheme and deduping enabled, the track should have a ref count of 5 on the empeg. one day, i want to delete one of the 5 albums that contain the track. i hit delete on the playlist, and it when it confirms, i check the "remove completely" tab, and hit yes to all. what happens is that the other 4 remaining albums will be nsync-less. what i want to happen is that jemplode checks if the refs is 0 before removing completely, to not break up my scheme. its similar to hard links in linux. it only really deletes after the last reference is gone.

the only autosyncing that RMML does is in the sense that when you start making changes, it will automatically begin to sync in the background ... It doesn't support any of the sync profiles or anything -- is that what you're asking about?

yeah, i was hoping that there was more to it, like being able to mirror a directory structure, with additions and deletions made on the pc reflected on the empeg. got the hashes now. pretty easy to go:
if empeg->track->hash doesnt appear on the pc anymore, then delete on empeg.
if pc->track->hash doesnt appear on empeg, then transfer to corresponding playlist.

question. directed also to mlord.

is hijack v345 still necessary since this new jemplode is out? will it hurt to have it on? i'm sticking with v344 for the reason that i'm afraid its gonna break something.

another anomoly. not sure if its on purpose. when you sort your playlist by artist, it discounts "The". see attached picture

Necessary? No. Very very desireable? Yes. Especially if you use fid subdirs to speed up your player and want to guarantee that they don't get messed up.

Cheers

Okay, here's my pet feature request:

I want a "fast sync" option, where JEmplode does the database rebuild locally on the user's PC, based on the previous database + any changes that were made. It then just FTPs the new database to the player, so that the player doesn't have to take 2-10 minutes to rebuild it from scratch.

An option, of course.

Cheers

In reply to:

---

and hit yes to all. what happens is that the other 4 remaining albums will be nsync-less. what i want to happen is that jemplode checks if the refs is 0 before removing completely, to not break up my scheme.

---

So this sounds like just regular "remove" to me ... The whole point of remove completely is to remove all references (so by definition, refcount > 0). What is the problem in this circumstance with not checking remove completely/

I've considered writing this ... Does anyone know exactly what steps are required to do this properly? I know what files I download to retrieve the database -- it is as simple as just replacing those couple files?

jEmplode should work with both -- it attempts the new way first, then falls back to the old way (well, not the way old way, but the middle old way )

ms

This is intentional ... The, A, and An get switched to the back (The Birds sorts as "Birds, The"). This was generally the most popular sort choice.

ms

And restarting the player after. At least I recall manually tweaking a database with the 2.x software that way.

The problem, though, is a database format change then becomes your problem.

So this sounds like just regular "remove" to me ... The whole point of remove completely is to remove all references (so by definition, refcount > 0). What is the problem in this circumstance with not checking remove completely/

well yeah. all i have to do is delete without removing completely, and then search for refs=0 and delete whats not in a playlist.

i just wanted to to take out these steps. just a pet peeve that i'm sure no one else will reflect. just ignore me. =)

I see now ... What you want is not necessarily to modify remove completely, but rather you want to bring back the old Empeg rules where when the refcount goes to 0, the tune gets removed. So are all your tunes in at least one playlist? If so, I could maybe bring back a configuration option that would make it so when you do a delete it will check refcount and actually delete the refs = 0 ones ...

Hi Mike, are you going to release the source for 46 or wait until another version to do so? I still would like to write my Palantir PDB Creator code so it would work with JEmplode.

I believe JEmplode *already* is dependent on database format -- it pulls down the current database files from the player and has to know how to parse them already. So no biggie going the other way.

And yes, it's as simple as that -- the database, playlist, and tags files.

Cheers

Mark, does this mean that there is no need for the players 200mhz processor to do the database rebuilds? A PC could just do it and send the results? As long as Jemplode could recover and "re-send" if there was ever a power loss to the player or a connection drop, that'd be really cool.

So cutting the database rebuild time in half wasn't enough for you huh?

The 200Mhz SA CPU is not the limitation here -- it sits practically idle while the rebuild goes on -- waiting for disk I/O nearly 100% of the time.

You see, the player software is very conservative about how it rebuilds the database, because it doesn't know what has or may have changed since the previous database was built. So, it discards everything (or is told to do so by (J)Emplode), and rebuilds from scratch, re-reading every single *1 tag file from the fids directories. On a 10000 song player, that means it has to do at least 10000 disk seeks and 10000 disk reads, plus all of the directory lookups and metadata reads required to locate the data blocks for those 10000 tag files.

Whereas JEmplode, knows what changes it is making, and can simply apply them as a delta to the previous database file from the player, without reprocessing 10000 tag files in the process. This will work fine nearly 100% of the time. And for a really screwed up player, we still have the option of letting the player regenerate it all from scratch if need be.

Cheers

Ah... . very interesting. Any idea why this isn't part of emplode already? Is it because it works "nearly 100%" of the time and not "each and everytime"?

Either way, very cool idea!

Yeah -- I can .. I was thinking I'd release it for 47 after I fix some of these bugs, but I can put 46 up if you'd like. Incidentally, you can do plugins now for jEmplode, so just by implementing a simple interface and making a special file in your plugin jar file, it will be detected and added to the Plugins menu. I can put a sample plugin file up for you.

I would guess that Emplode lacks it because, when Emplode was first written, a BIG empeg player had like, 4GB (!!!!!) of drive space, and a database rebuild was probably only a few seconds even for a full player. So no biggie.

Things change..

And even if I have to rebuild the entire thing (without deltas), I have them all in memory, so it's pretty painless ...

ms

Mike, that would be awesome. I'll begin writing the plugin for Palantir as soon as you get it up somewhere for me to download!

Are you talking about an auto install for Palantir or an auto export of the database? Either way: Cool!

Auto generation of the .pdb file you need to load on your Palm each time you add or remove songs. Depends on what options are available to me, but I could make it auto-generate when syncing, and then launch the .pdb file to make it install on the Palm. Otherwise, it would be available as a menu option (Plugins -> Generate Palantir PDB).

An automated install of Empire is waiting until we get a "site exec" command in Hijack.

Any idea why this isn't part of emplode already?

1) What Mark said.
2) emplode hasn't seen any development effort since 2.0-final was released. When I left, it was planned to include v3.0 car player support in the next version of Rio Music Manager, since it's the most flexible platform, going forward, but that'll take some time -- not least because there's a lot to do -- I'm guessing.

For the simple plugin docs:
http://www.inzyme.com/rio/plugin.html

In your case, I would recommend making your plugin action extend org.jempeg.music.manager.action.ExportAction and just call the superclass passing in your custom exporter impl.

ms

Hello Mike,

I have a short question. Why I am not able to upgrade to the latest Hjjack version anymore. I did compare the settings with jEmplode 45. I have the same setting, but 46 tells me everytime, that I have to have the Hijack 242 or higher installed, connected over ethernet and "Use Hijack when possible. I fullfil all the requirements and I can download with jemplode 45 the latest Hijack with the same settings on a other computer.

Thanks a lot

I'll take a look at this ... I maybe hosed it up.

ok...Cool...

Thanks for the very fast response..

bye

Hey Mike, do you have Javadoc available for this release of Jemplode? I wanted to poke through it and see a good place to start to get the node information for each song.

This version moved directories ... I haven't had as much time to work on jEmplode this week as I thought I would. I'm hoping I will have time this weekend, though, so I can address some of the bugs found.

http://www.inzyme.com/rio/empeg = Empeg stuff
http://www.inzyme.com/rio/empeg/install.htm = jEmplode 46 Installer
http://www.inzyme.com/rio/empeg/jemplode-src.zip = jEmplode 46 Source
http://www.inzyme.com/rio/empeg/docs = jEmplode docs
http://www.inzyme.com/rio/empeg/docs/javadoc.zip = zip'ed javadocs
http://www.inzyme.com/rio/empeg/docs/api = online javadoc
http://www.inzyme.com/rio/empeg/docs/plugin.html = plugin docs

Just responded at the top of this thread with javadoc and src location ... If you grab the source, there are several examples of IExporter implementations that will show you how to enumerate all the nodes in the database. Roughly, your plugin has an EmplodeContext, from which it can get a PlayerDatabase, from which it can get an FIDNodeMap, from which it can get an enumeration of all IFIDNodes in the system. Those nodes will each be one of three types -- FIDLocalFile (for a recently imported and not uploaded tune), FIDPlaylist (for a playlist), or FIDRemoteTune (for a file on the Empeg). From an IFIDNode, you can get a NodeTags, which is basically a hashtable-interface-with-types. The valid tag key names are all defined as constants in DatabaseTags.

ms

Mike, you rock. Thanks.

Very nice. I was using whatever the old version was and was going to come post about how it couldn't read iTunes track numbers. Realizing that there was no point in posting bugs for old versions, I downloaded the new one, and it worked perfectly. Now to see what else has changed...

Matthew

In reply to:

In reply to:

tags changed on the hdd doesnt change the tags on the player if it recognizes as a dupe.

yeah .. this is a known limitation (of emplode and RMM too, I believe). this is, in general, a hard problem because you can change tags independently in each location, so there's no way to know (automatically) which direction they should sync.

Check timestamps (mtime) on the respective files?

I have been playing with the new version and it seems pretty slick. The background syncing and dedupping are great. The only weird thing I noticed so far is it that it seems to be quite a bit slower to upload songs. According to netstat I am getting around 400k/sec and I think the older version got closer to 700k/s. Did the old version use hijack's ftp server and the new version doesn't? I have both Use Hijack and Use Fast Connections selected in the prefs.

-Mike

I've never actually supported hijack for uploads (only downloads), though I've thought about adding hijack upload support. However, fast connections is /supposed/ to be pretty comparable in speed ... It's possible fast connections aren't working right now (now I'm wondering if I pulled that code out because there isn't an equivalent concept on the Karma -- I bet I did). I'll take a look at that.

http://www.inzyme.com/rio/empeg/jemplode-src.zip = jEmplode 46 Source

Is it just me or does this not compile? I wanted to poke around a bit, but org.jempeg.empeg.music.tags.* seems to be missing.

-Mike

it's not in the jempeg-empeg project under jempeg-empeg/src/org/jempeg/empeg/music/tags? If not, then something funky must have gone down when I zipped it up ...

it's not in the jempeg-empeg project under jempeg-empeg/src/org/jempeg/empeg/music/tags?

Nope, that whole directory seems to be missing.

-Mike

I've never actually supported hijack for uploads (only downloads), though I've thought about adding hijack upload support. However, fast connections is /supposed/ to be pretty comparable in speed ... It's possible fast connections aren't working right now (now I'm wondering if I pulled that code out because there isn't an equivalent concept on the Karma -- I bet I did). I'll take a look at that.

It looks to me like you still have fast connection support in (unless I am missing something obvious). Anyway, I modified v46 to support fid uploading and deleting via ftp and I am getting around 700k/sec (that is to an ext3 filesystem, ext2 may be faster). Also, the small file operations (uploading fid text files and deleting fids) are significantly faster using ftp directly. Anyway, let me know if you are interested in a patch and I will finish it up (right now I have it uploading everything to drive0, gotta make that more intelligent) and send over a patch.

-Mike

Yeah -- absolutely ... Thanks!

ms

around 700k/sec

Hah, I just realized when I was testing that I was on my wireless network. On the wired net it looks more like 800-900k/s. That is definitely faster than I ever got out of the empeg protocol.

-Mike

I barely understand half of what you guys are talking about, but it's exciting to see you two working on this!

Hey Mike is there a "right way" to tell the player in the middle of a sync that I am not going to be sending data to it for a while? It seems to time out and restart if you don't send any commands for a while (while ftping a large file). I could call EmpegProtocolClient.isDeviceConnected() occasionally to keep its attention, but that seems kind of weird.

Kind of weird situation ... isDeviceConnected() is basically implemented as a ping(), so it's probably your least invasive route. Probably the approach I would take ... I agree it seems a little dirty, but we're basically bypassing Empeg's synchronization, which is kind of dirty by definition

Don't give me any credit ... This is all Mike ...

With all of the transfers being done via ftp, does that mean that the player would continue to play it's software until the database rebuild kicks in?

With all of the transfers being done via ftp, does that mean that the player would continue to play it's software until the database rebuild kicks in?

And with jemplode rebuilding the database, would there be any need to do anything besides reboot the player to see the new music?

The synchronising screen does serve a purpose in communicating that it's the one time you can't unplug your player, however.

Matthew

With all of the transfers being done via ftp, does that mean that the player would continue to play it's software until the database rebuild kicks in?

Ummm, I think it would be possible to delay the sync until the DB needs to be rebuilt but I wasn't planning on making that drastic of changes to jEmplode. I don't want to break any of the existing stuff so I am only modifying the routines for uploading songs and deleting songs. Maybe later I will think about it....

-Mike

And with jemplode rebuilding the database, would there be any need to do anything besides reboot the player to see the new music?

Now the really sexy thing to do would be to upload everything via ftp and rebuild the database on the PC running jemplode. Then all that would have to be done is uploading the DB to the empeg and rebooting it. Then the empeg would never really have to go into its syncing state. That is beyond my understanding of how jEmplode works at this point though.

-Mke

I thought of that just after I made my post. I remember talk of database rebuilds being done on the PC in another thread. The only issue I could see is if you deleted a song and then a moment later, the empeg tried to play it. Wouldn't that cause the player to hang?

Well, you could just do the deletes after the new database is loaded.