jEmplode 32 - Fast Uploads! Animted GIF Import!

http://www.jempeg.org/jemplode20.jar

Here's the skinny --- The big new features are Fast TCP uploads (Thanks Peter and Hugo!) that give a 50-100% speed improvement on uploads (Hey Tony -- try this on the Jupiter!). Also, AnimEdit can import animated GIFs now (it's up to you to keep the palette correct!!) -- it will even do frame uniqueing, so if you have any duplicate frames, they'll end up sequenced properly. There are a whole bunch of other misc tweaks and optimizations (see below)... Keep in mind that with agressive optimizations typically come some weirdness -- I tried to work all of them out, but just keep an eye out for any strange behavior (specifically relating to the playlist tree updating as you fiddle with the table view -- I spent a lot of time trying to get updates to propogate faster)

1) jEmplode - reorder is fixed
2) jEmplode - post-reorder selection is fixed
3) jEmplode - removed a bunch of inefficiencies with copying Playlist Nodes
4) jEmplode - added mnemonics to menus
5) jEmplode - fixed goofiness with selection + sorted columns
6) jEmplode - added Ctrl-Insert,Shift-Insert, and Ctrl-Delete Accelerators
7) jEmplode - fixed the jumpy reorder
8) jEmplode - fixed selection after pasting into soup
9) jEmplode - changed repaired errors message
10) jEmplode - fast connection support (1.5x-2.0x the upload speed!)
11) AnimEdit - load animated GIFs (collapses duplicate frames for you!)

oh.. YES... i love you mike.. i wanna buy u lunch sometime.. man now i need to get moving on these animations.. WOOOOHOOOO!!!

So awesome. I so want to see them on the Peg, especially Tony's risky effort . I am using Hijack v246, Jemplode v32. I generally use the exe, thought I've tried the straight .jar. I am using J2re 1.4.0.
When i try and use the animation editor to download or upload an image, I get the message saying : use v242 or higher, hijack when possible, connect via ethernet.

The hijack when poss is ticked, I have no problems synching or using anything with jemplode over ethernet. So I meet the criteria, near as I can tell but still get the message. I have tried moving back and forth jemplode versions, and hijack versions, and checked my config.ini for funnies.... still no go. Since no one else seems to be having this problem, something must be different...

Any thoughts?

Cheers,

Kieran
OZ.

And you're actually connecting to your Empeg with Ethernet w/ jEmplode when you run the Animation Editor? (i.e. 1) start jEmplode, 2) choose your Empeg, 3) Connect, 4) Tools..Animation Editor?)

Mike

Hey Mike I haven't had a chance to try 32 yet but are the drag-and-drop import problems fixed by any chance? I was still having problems with it as of 31.

I just tried it here and it seems to be working fine ... What OS are you doing it on? I think Linux maybe had problems with DND (the OS just delivered a blank list to Java). I'm using Win32 and OS X and they work (I don't run X here, just a headless Linux). Grab 32 and give it a try (there were a lot of little fixes). Send me any errors that appear on your console.

Mike

I was on Windows 2000. I will try another tonight when I get home. Actually I have to, because my player application seg faulted last night during an Emplode sync. Hopefully my luck is better tonight.

Yeah -- definitely try again. It should absolutely work on Win2k (that's what I use at home).

if it fails, go into jEmplode2.lax in your install directory, and look for a line that says:
lax.stdout.redirect=something
set it to
lax.stdout.redirect=C:\\debug.txt
(notice double backslash)
then add a new line
debuglevel=56
then run the app, drag and drop, quit, and send me the output of debug.txt

Mike

Mike, you've done a great job on the order-tweaking code! Thanks for doing all that stuff.

Odd things happen when you move up/down when you're sorting by (e.g.) Title, as the two items are swapped, disrupting the sort order. But perhaps that's better than the old behaviour where nothing moved, and the only sign of any action is that the Position field changes. Maybe JEmplode should automatically switch to "sort by Position (ascending)" as soon as you touch the order buttons, if not already in that mode? Or just refuse? I don't know the best way to deal with this.

But that question aside, v32 is a great improvement on v31 in this respect. Thank you.

hehe .. pretty funny behavior... I have _NO_ idea what it's doing there It's kind of funny though (it looks totally random? anyone who can explain this wins a cookie). I like the idea of auto-switching to sort-by-position when you hit the reorder keys ... would that be more or less confusing? (well, than the "just changes the position field" one -- obviously it's less confusing than the current behavior)

Mike

The current behaviour makes sense if you think of what it's doing in Position-order mode:

Suppose you press CTRL-UP. Then item n is swapped with item (n-1) and these two items are swapped in the display - without moving them. Er, it's quite hard to describe!

Try it a few times, looking at the Position field. The selected item will be swapped with the item that has position=n-1. This is obviously the Right Thing when the items are in Position order, but it looks weird otherwise...

Do I get the cookie? Or is that too poor an explanation?

(Hey Tony -- try this on the Jupiter!)

Did so.

Uploading a single song worked the first time I tried it, but it seemed to be slightly slower than I expected it to be.

So I deleted the song from the Jupiter, then sent the same song again with emplode and timed it.

Then I deleted the song again and tried uploading it with Jemplode for a timed test. And it won't upload now. At all. It stalls out completely. I've tried several times now, emplode always sends the song fine, Jemplode always stalls.

Attached is the status.txt file.

Oh wow -- Good job ... http://www.the-dma.org/consumers/graphics/cookie.gif

Mike

Hmm .. I wonder if it's possible that Jupiter doesn't support fast upload ? Peter/Hugo/Roger? It's also possible I just hosed up the implementation ...

Two cookies! (One in the HTTP header and one in the image). What a lucky boy I am!

Okay this database stuff with jEmplode is getting a little crazy. I previously reported that I was getting warnings about something wrong with my DB. Well I upgraded to 32 and today instead of a warning I got "1 errors were fixed. Synchronize to save the fixes." Emplode isn't finding anything wrong with it... Is jEmplode's database checking somehow stricter than Emplode's? After turning debug up I see the line "There was no tune or playlist that has ID 272" and I think that might be due to the fact that an Emplode sync crashed and I had no recourse but to reboot while my drives were still RW. I'm still assessing whether any damage was done... But why wouldn't Emplode find this too?

By the way I did the drag and drop again with debug turned up and I'm getting an OutOfMemory error:

1016664710723: Recursing playlist repair/check: Soul (fid 29792)

1016664710723: Finished recursion of playlist Soul (fid 29792)

1016664710723: Finished recursion of playlist Urban (fid 26192)

1016664710723: Finished recursion of playlist Singles (fid 29968)

1016664710723: Finished recursion of playlist All Music (fid 256)

1016664710723: There was no tune or playlist that has ID 272

1016664710723: Finished tree->Repair() recursion

1016664710723: Checking reference counts

1016664710723: Finished checking reference counts

java.lang.OutOfMemoryError

	<<no stack trace available>>

.

Any idea? I've got 256 MB physical and some 512MB swap, task manager reports a boatload of free memory... Is there a way to give the JVM more memory to play with, or does it automatically take what it needs to take?

Sigh. My next step is to try JDK 1.4 and see how things roll with that. I'll keep you posted.

Update:
Well I installed JVM 1.4, rebooted, and same behavior. The folder I'm drag-and-dropping contains about 570 MB, but it's not like it should be allocating 570MB of RAM at once... Right?

I wonder if I'm holding onto memory somewhere ...Never seen that before. About how many songs tunes are you dragging-and-dropping?

jEmplode is ported off of emptool which does have more strict considerations for errors (things like 0 reference count = error in emptool, but = soup-only song in emplode) -- It's possible that since missing references can be automatically fixed that Emplode doesn't actually consider it an error (emptool does though). You're seeing the new error message (the old one would have incorrectly told you to look in Unattached Items). Same error, different message. After you did a sync, did the problem go away? Usually if I get those (which are exactly what you said -- dropping connection during a sync causes those), then one sync (with Emplode _or_ jemplode) will fix it all up. Let me know.

I'll look into the memory thing.

Mike

You can allocate more RAM to the JDK, by the way, by running java -Xmx128M -jar jemplode20.jar for 128M of ram... I think it defaults to only using up to 32M? But still, that sucks .. I don't know why it's falling over like that... Try raising the RAM to 64M and see what happens while I go on my hunting trip.

Mike

137 tunes being dropped from a folder which already exists. Theoretically it should just add the new songs right?

Oh, sweet, that's almost how we add more RAM to our iPlanet Application Servers at work. I think the switch is -ms128M for more stack and -mh128M for more heap.

Anyway I just tried it and no dice. "VM Size" in task manager shoots up from about 35 MB to over 145MB and the OutOfMemory error is thrown.

Yeah -ms128M is the JDK 1.1 way to set heap size .. -Xmx128M is the 1.2 and above way to do it. OK so this is definitely a bug .. I'll see if I can reproduce here.

Mike

OK, Now I'm curious ... I do the same test with several hundred MP3's on my Win2k box and i barely jump up memory usage ... Hmm... Any particularly large individual files? I don't think that matters, just curious ... Everything should be stream-based, so I can't imagine what could be going on.

Grab http://www.batmanthehorse.com/jemplode/jemplode20.jar and try it out for me .. This is using a different MP3 tag parser library.

Mike

Well whatever that new JAR file has is working beautifully now. No problems at all.

I was afraid of that ... Apparently there is a memory leak in Dan's MP3 parser. I might enable this alternative one (which seems to have a higher success rate anyway) for now and email Dan.

Mike

Okay I'm going to go right from bug report mode to unreasonable feature request mode

So I do my import of my root playlist and it tells you which ones were duplicates but doesn't tell you which ones were successfully imported. Maybe have two listboxes, one with the ones that were OK, the other with rejections? I know the playlists with new items are in red, but it'd be nice to have a quick summary so that you know everything you expected to get imported went through okay.

Any drawbacks to this MP3 library by the way?

No drawbacks to this MP3 library.

All the "success" data is there, I just don't display it ... Other people didn't like seeing "everything worked fine" messages (I tend to like it, myself). Maybe I'll make it a configuration option.

Mike

Hey, is it me or does Set Playlist Order not work? Is that implemented yet?

Edit: Nevermind I figured it out. Just doesn't work the same way as Emplode... Better actually.

Oh, and Java 1.4 doesn't seem that much "snappier" to me. Even with Java being as platform-independent as it is, I can't understand why Java apps feel slow on my Duron 1000 system. Mind boggling.

Really? How does Emplode work?

1.4 should be faster for things like scrolling in the playlist view. I definitely feel the speed difference in the 1.4 demos ... But I really haven't done much work at all on GUI performance -- I'm sure there are all manner of crazy inefficiencies going on in poor jEmplode.

I agree that Java still has a long way to go ... Think of it this way, though -- Since you're running in a VM, you're basically doing the last few stages of traditional compiling on-the-fly (generation of native code and linking). Still kind of sad though

Mike

Wow, i just did a quick upload test and it's noticibly faster. ROCK!

[edit]
Jemplode - 54.2megs of songs - 1min:28sec
emplode - 54.2megs of songs - 1min:20sec

There goes that theory.

He didn't mean faster than emplode. He meant faster than previous releases of Jemplode.

Ahhh... i totally got the other impression. doh.

Ok... Figured out prob from 3 post reply from the top of this thread.... That was only yesterday... and look at this thread now..... no sleep for the wicked....

Problem getting animation editor to connect at all, but rest of emplode ok etc...

I have a kftpd_password=heytheretux set in my config.ini.
Remove it, reboot the peg, problem goes away.

Also when the pwd is set it also made -update hijack- tells me everything was upto date, even when I went back hijack versions....

I need it set because multiple people at work stream mp3's and video from it, and I'don't want to have to switch to insecure @home profile, whilst still connected to the LAN, or wait until I get home to put the latest risky or bond animation on my player

Can you do third profiles? @update_animation ? I forget....

Getting rid of ;@WORK khttpd_files=0 temoprarily which currently prevents updating (and may prevent animation connect - haven't tried it yet), is less of a security risk than opening up FTP to the peg in my book.


Wierd that with all these people creating animations, no-one had this problem yet.... I guess they either have @home's without ftp pwds (I sometimes share mine over internet and vpn's from home, so still have it set), or people aren't doing too much streaming at work or whatever....

Would someone like to have a quick go, and make sure I am not talking absolute pants?

Is it just a matter of catering for ftp passwords read from config.ini?

Cheers,

Kieran
OZ

Whoops .. yeah ... I never added and user or password support into jEmplode

I've had this problem and I posted about it a while ago...
hopefully Mark can do something to allow check the /proc/version file even while files=0
the issue with files is bsically moot if you turn off your password, but if it's on and files is not set then you can view the config file through http and get the ftp password out of it...
that would be bad

I've noticed something similar happen when I select a bunch of tracks in a playlist that's sorted by Source and edit their properties to change the Source... I imagine it happens for other fields, too.

Apparently there is a memory leak in Dan's MP3 parser

I have gotten out of mem errors importing mp3s with id3v2 image tags before. I am not sure if it was with DMZ's taglib or the older id3lib based one though. I bet if Tony uses something to strip out any less common tags it will work with the newer parser. I eventually wrote a perl script to strip and clean my tags on all my mp3s which is the main reason I never have bug reports for you any more :-)

-Mike

Yeah -- we started with that id3lib port (which sucked), then Dan donated his (which was much better), and currently (temporarily, if Dan has time to work on his -- I know he's busy these days) we use JD3Lib (http://sourceforge.net/projects/jd3lib/) which seems to work really well (and also parses VBRI header information so we can use it for that to).

Mike

and also parses VBRI header

Not properly, it doesn't . I was hoping that it'd give me some pointers to parsing the TOC properly. Unfortunately, it doesn't do anything with it. This is unfortunate.

It does, however, get the average bitrate, etc. correct, but emplode can already do that for VBRI files.

The guy who wrote it is pretty helpful -- I'll drop him an email and ask if he is planning on adding any code for that.

Mike

Here's what I turned up in a search:

toc (table of contents) gives seek points for random access
the ith entry determines the seek point for i-percent duration seek point in bytes = (toc/256.0) * total_bitstream_bytes
e.g. half duration seek point = (toc[50]/256.0) * total_bitstream_bytes