here's what i've discovered about the windows version. if i modify a file using winamp (or anything else for that manner), mp3tofid immediately recognizes it as a new file (i see 1 empty fids in the summary screen), and deletes the old fid. not sure why.

This all depends on how your id3 tag editor works. If it writes out a new file, deletes the old one and renames the new file to the old name, then you have created a new file with a new inode number. If it just opens the file in read/write mode and makes the changes inside the file, then the inode number remains the same.

i dont understand whats happening in rsync i guess

You can increase rsync's verbosity by placing the --verbose on the command line multiple times.

Pim

Thanks for your suggestion. I added the major/minor stuff in the lookup key as to enable support for cross-filesystem symlinks and mountpoints within your mp3 tree.

It did not occur to me that major/minor numbers could change at all until I read your report.

Pim

i've been noticing that some mp3s that i have in the collection shows as invalid format when played. i decided to erase and re-upload using emplode, and it worked fine. could a bad parse of the tag cause this?

also, have you noticed that when the rsync algorithm is used, it takes longer than just doing the whole file. I guess its just the weak processor of the empeg making the checksums thats slowing it down. If i edit an id3v2 tag, I get around 350k/sec regular, and 700k w/ the --whole-file switch.

If i edit an id3v2 tag, I get around 350k/sec regular, and 700k w/ the --whole-file switch

The problem with id3v2 tags is that they are at the beginning of the file, rather than at the end, like id3v1 tags. If you insert a id3v2 tag, or when its size increases, the mp3 frames are shifted to a different location in the file. As far as I can tell, rsync cannot detect this and compares the whole file, only to find out it has to transmit everything again, because none of the file parts are identical.

Pim

In reply to:

---

If you insert a id3v2 tag, or when its size increases, the mp3 frames are shifted to a different location in the file. As far as I can tell, rsync cannot detect this and compares the whole file, only to find out it has to transmit everything again, because none of the file parts are identical.

---

If that happens, your rsync isn't working correctly. The rsync protocol is designed to find common parts of files so that it can cope with this kind of thing - otherwise it would also fail on e.g. tar files when a file is added near the beginning. (Of course, compressed tar files are a different matter...)

so anyway, regarding the fact that some files are being considered invalid format... here are the two tag files that were generated for the same file... one by emplode and one for mp3tofid.

emplode
artist=New Energy
bitrate=fs192
codec=mp3
ctime=1037042767
duration=410540
length=9895462
offset=42492
samplerate=44100
title=Energetic Wave (Nu-Nrg Original Mix)
type=tune
year=2002

mp3tofid
type=tune
artist=New Energy
bitrate=fs192
codec=mp3
ctime=1034559218
duration=410514
frames=15715
length=9895462
loadfrom=/cygdrive/e/mp3/2002/1002/new_energy-energetic_wave_nu-nrg_original_mix.mp3
offset=1540
samplerate=44100
title=Energetic Wave (Nu-Nrg Original Mix)
year=2002

parsing error in mp3tofid?

You're right, I should have just tried it instead of trying to understand the sources.

Pim

here's an anomoly. try to just change the datestamp of one of the mp3s. when you rsync, it still appears to transmit the whole file. nothing changed at all, except the stamp. i'm going to do a --verbose 4 times and capture the log. hopefully, you can understand whats happening.

offset=42492

That's it. mp3tofid assumes it can find the start of the first mp3 frame in the first chunk it reads.
The chunksize is 40960 (#define INPUT_BUFFER_SIZE (5*8192)) in scanmp3.c.

Your mp3 seems to have a much larger id3v2 tag than I then the ones I tested with.

The lazy workaround to this bug is to increase the buffersize and recompile.

Unfortunately I will not be able to help, as I am packing for a three week trip to Asia.

Pim

If timestamp and/or size are different on each side, rsync has to check checksums of each side. This might be very slow if the empeg is involved, like you have seen in the case of editing id3 tags.

Pim

but the point is, emplode saw the offset at 1540. its the same track so... in theory, shouldn't all the parameters found be the same in both tag files?

edit: whoops. i'm dumb. it was mp3tofid that saw the offset at 1540. i have found that the files that were failing were because of un-standard hidden id3v2 tags, which i promptly erased. everything is fine now.

It's a bug nonetheless. I'll try to fix it when I return.

The other notable differences are seen in ctime and duration. The ctimes are inherently different, as mp3tofid uses the ctime of the source file, while emplode uses the ctime of the fid file on the player. The calculated duration is different too, but then I have yet to see two programs that come up with the same duration in milliseconds. I could be wrong, and libmad could be wrong, but emplode might as well be wrong. Then again, the player copes well with wrong duration tags, as I have seen during development, when mp3tofid durations were way off.

But beware of different offset or length. They have to match exactly.

Pim

i've found a bug in your compile of rsync. it seems to have the rolling checksum broken. with the --stats switch, if you only transfer one file, Matched data:will always be the same size as the block-size. even if there is no difference in the file. not really sure if its just a display error or not.

You could try the debian version (2.3.2) to see whether there's any difference.
My experience with the debian version is that it locks up when transferring/comparing huge numbers of tunes (>14000 in my case).

Of course, the problem you are seeing might also be caused by cygwin's rsync.
For testing, it would be nice to have a replacement for that too.

I'd be interested to see what you have found out. Now's the time to shut down my computers and leave for the airport.

Pim

It's been working great since. I've got a feature suggestion for when you get back from your trip though - I've got some music (holiday music) that I really don't want on my empeg. Can I suggest either a command line switch to exclude a directory(s) from mp3tofid scanning, or maybe a flag file we could place in a directory that would signal mp3tofid to ignore the directory (Maybe ".no_empeg" or something)? Either would be nice, I can see plusses and minuses for either solution. Thanks..

Hi.

Couldn't you use an md5 hash over the mountpoint _name_ instead of the partitions major/minor? By that, mp3tofid wouldn't recreate the whole database if the drive and/or partition changes its position, which can happen quite often on some systems (like mine: I often add/remove harddiscs or controllers, but the mount point for partitions always stays the same because I mount them by volume labels).

cu,
sven

i've found a bug in your compile of rsync. it seems to have the rolling checksum broken.

Yes, it appears rsync 2.5.5 on the empeg will always transfer a file, even if only the timestamp is different. This is a bug that supposedly was in 2.5.4 but has been fixed in 2.5.5. But the problem also appears to be there in version 2.4.6, which can be found here. It's also there in the latest CVS version.

The only version that does not appear to have this bug is the 2.3.2 debian version. Unfortunately, this version has other problems.

I did all my testing against a linux x86 box with rsync 2.5.5.

I'm not sure what causes this bug. Maybe an ARM alignment problem, or a gcc bug. I can't find anything in the rsync mailing archives either.

Pim

Couldn't you use an md5 hash over the mountpoint _name_ instead of the partitions major/minor?

I don't think there's a fast and portable way to discover the mountpoint name. major and minor numbers are easy, though, as they are returned in the stat() data.

Thinking about it, I might just as well have ommitted the mountpoint data, unless it is different from the mountpoint data of the root of the mp3 tree. This would solve almost anyones problems, and it would still work in the rare cases where mountpoints exist within the mp3 tree.

Pim

welcome back from asia btw. how was the trip?

as for rsync... i'm content w/ 2.3.2. don't really see a problem with it, and i was able to speed up the upload of a slightly modified mp3 file (id3 tag mods) by putting the checksum block size to 100000 bytes. it minimizes the transmittion of checksums/packets.

i was wondering if it would be possible to "import" a current fid/file structure on the empeg. there is a bug that got me by surprise. mp3tofid segfaults when you change operating systems (xp to 2k in my case). i deleted everything EXCEPT the mp3tofid association file, and still segfaulted. finally gave up, deleted the mp3tofid file, and started anew. but my point is... if you took all the tag files (______1) from the empeg, couldnt you parse them and rebuild the associations, depending on the fact that the import_location field is filled (mp3tofid and jemplode use that field and emplode seems to have omitted it). the only limitation is in windows based systems that used jemlode, you'd have to convert "c:\" to "/cygdrive/c".

speaking of jemplode, i'd really like to be able to have the hash value of an mp3 calculated and put into the tag file just like jemplode. i know that you don't use jemplode... but i use it to do the stuff that mp3tofid can't do (yet).

welcome back from asia btw. how was the trip?

Thanks, I had a great time visting Bangkok, Hong Kong, Macau and Guangzhou.

as for rsync... i'm content w/ 2.3.2.

Lucky you. Mine kept locking up. Not during large fresh transfers, but during small changes inside 70 gigabyte of MP3's.

i was able to speed up the upload of a slightly modified mp3 file (id3 tag mods) by putting the checksum block size to 100000 bytes. it minimizes the transmittion of checksums/packets

You'll be transferring at least one 100.000 byte block then. Did you time different sizes as well?

i was wondering if it would be possible to "import" a current fid/file structure on the empeg

That's going to be hard, as plain empeg fids do not contain the "loadfrom" tag.

mp3tofid segfaults when you change operating systems (xp to 2k in my case)

mp3tofid should not segfault, but you lose anyway. mp3tofid since version 3 uses not only the inode number, but also the major and minor number of the filesystem to uniquely identify a file. Doing so, mp3tofid will handle the rare case of mountpoints within the mp3 tree, which could allow for duplicate inode numbers.

If you copy your mp3's to a different computer, every mp3 file will probably have different inode numbers and the filesystem containing them will probably have different major/minor device numbers. So all fid numbers will be invalid, and will have to be recalculated. And new fid numbers means yet again a new whole rsync transfer. Sorry, but this is how it's designed.

Pim

i'd really like to be able to have the hash value of an mp3 calculated and put into the tag file just like jemplode

Is that a plain md5sum hash? Should be doable, as mp3tofid reads the whole file anyway.

Pim

You'll be transferring at least one 100.000 byte block then. Did you time different sizes as well?


i tried a 500,000 block, but it seemed to take longer. didn't really want to to test it too much. i'll try it tonight.


That's going to be hard, as plain empeg fids do not contain the "loadfrom" tag.

If you copy your mp3's to a different computer, every mp3 file will probably have different inode numbers and the filesystem containing them will probably have different major/minor device numbers


i know that the plain empeg doesn't use the tag. but the tags already generated have them, as well as jemplode. to recover from an inode change, can't you parse the existing tags, find out the new inode/major/minor#, and repopulate the mp3tofid file, assuming that the file structure was the same.... (or just mass replace the path to wherever the files are).

Is that a plain md5sum hash? Should be doable, as mp3tofid reads the whole file anyway.

its a crc32 check on only the mp3. exclude the tags.

is there anyway aside from hitting q in terminal mode to kill the player? i've tried to just use kill -9 pid but it always restarts.

You can gracefully stop the player with "kill -INT <player-pid>"
and restart it with "kill -HUP <bash-pid>". Or use killall from
debian to automatically find the pid.

Another way to stop and start the player is to use hijack:
http://player-ip/?NODATA&SERIAL=q will stop the player,
http://player-ip/?NODATA&SERIAL=exit will start it.

You would need wget or something to do this from the player,
but there might be a more direct way to have hijack emulate
serial input. Anyone?

Pim

Yeah. You should be able to:

echo "q" > /proc/empeg_notify

echo "q" > /proc/empeg_notify]

Hmm... Does that work for you? Not for me, running
hijack v310. Would be cool, though.

Pim

EDIT: Oops, forgot to add

[output] 

notify=1

to config.ini

I didn't know that was a requirement. It's how empegVNC sends button presses to the player.

Well, it did not work, I made a post about it and then read the FAQ.

Wrong order ...

But now it appears not to work even with the mentioned addition
to config.ini. Did you try this? Of couse, you need to install emptelnetd
to try.

Pim

Okay, no, I didn't try it.

All I know is that it works for button presses, et al. I'd try it here, but I can't attach my empeg to the network at work.