Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 1 of 2 1 2 >
Topic Options
#120600 - 13/10/2002 16:44 mp3tofid 3.00 - now for Windows too
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
I have just released mp3tofid version 3.00 , to be found here as usual.

There are three major new features
  • support for (and defaulting to) the new player directory structure, ie. fids named /driveX/fids/_YYYYY/ZZZ. Use "-o" to revert to the old structure.
  • intermediate database for storing inode to fid relationship, resulting in a much smaller player database
  • a Windows port (actually a cygwin port)


Until now, it seemed mp3tofid only worked for me. Previous versions seemed to produce much too large databases for the player to handle. The database size would depend on how inode numbers were distributed on the filesystem containing the source MP3's. In this version, this should no longer be of influence.

The intermediate database is stored as .../drive0/var/mp3tofid. Don't lose this database. A new one will be created if it is lost, but all tunes will get new fid numbers resulting in a total re-upload if you run rsync again.

As a result of its location, the mp3tofid database will be copied to the player. It serves no purpose on the player, but it might be handy to have it stored as a backup there.

Pim

Top
#120601 - 13/10/2002 19:11 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
dmuench
stranger

Registered: 15/01/2002
Posts: 37
Loc: Honeoye Falls, NY
So far, it looks great. My 600 meg database has dropped to 229k. I haven't synced it all to my empeg yet, I'll try that tomorrow. Thanks again.

Top
#120602 - 14/10/2002 21:18 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
i'm getting this error when using rsync
skipping non-regular file "drive0/fids/_00000/8f0"

any help?

Top
#120603 - 15/10/2002 02:46 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
It's probably a dangling symlink. Have you checked with ls -l?

You can also use symlinks (written by Mark Lord) to check for bad symlinks.

Another possibility is that the rsync server does not have permission to read the file the symlink is pointing to.
The rsync server usually runs as nobody on Unix or SYSTEM on Windows, so the target should be world-readable.

Pim

Top
#120604 - 15/10/2002 08:04 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
its because i ran mp3tofid w/ drive letters (e:/mp3 e:/empegfids)instead of using /cygdrive/e/. its been syncing since last night.

another question.... i'm looking at the fids directory.... theres a blank files that aren't connected to files everywhere. is that by design?



Top
#120605 - 15/10/2002 09:37 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
i ran mp3tofid w/ drive letters (e:/mp3 e:/empegfids)instead of using /cygdrive/e/.

Ok, now I understand you're using cygwin. I was not aware windows-type paths worked at all in cygwin ...

theres a blank files that aren't connected to files everywhere

You mean zero-sized files? There should not be any.

You'll see three types of files:
- tag files, ending with "1". These are small (unix) text files.
- playlists, ending with zero. These are small binary files.
- symlinks, ending with zero. They point to you mp3's.

symlinks can only be seen as such by cygwin apps. Native windows apps see them as shortcuts, but they are twisted. Dont try to edit the shortcut properties. True shortcuts do not appear as symlinks on cygwin, but as plain files with a .lnk extension.

Pim

Top
#120606 - 15/10/2002 18:12 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
ok, i was wondering what the tag files were. now i know. but this poses another problem. is there anyway for mp3tofid to re-create the tag files if i modify them on the hdd? this was the whole point of me getting this to work (aside from the knowledge =).

Top
#120607 - 16/10/2002 15:01 Re: mp3tofid 3.00 - now for Windows too [Re: image]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
ok, i was thinking this over today. if mp3tofid can check the modified date versus the last time mp3tofid was run, if its a match, delete the tag file and regenerate.

also, can there be a switch to check for id3v2 tags only? i see that mp3tofid is scanning my huge compilation cds wholy, and i was wondering why?

Top
#120608 - 16/10/2002 16:01 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
if mp3tofid can check the modified date versus the last time mp3tofid was run, if its a match, delete the tag file and regenerate

This is basically the algorithm:
1. read all existing fids into memory
2. for each fid, do numerous tests to see whether it still matches the source tune, otherwise delete from memory
3. scan the source mp3 tree, skipping all tunes that have valid fids in memory
4. completely empty the fids directories
5. regenerate all fids and databases from memory
6. set modification date of tune fids and tune tag fids to their source file's modification date (this speeds up rsync a lot)

also, can there be a switch to check for id3v2 tags only?

It is libid3tag's behaviour to look for id3v2 tags and drop back to id3v1 if no id3v2 tags at all are found.

i see that mp3tofid is scanning my huge compilation cds wholy, and i was wondering why?

I could not find a reliable way to calculate duration and bitrate from an mp3 using libmad, other than by decoding each frame header, essentially reading the whole file. It's only decoding the headers of the frame, not the frame itself, so this is an i/o bound operation. On my system, it took 110 minutes to scan 72 GB of MP3's. (It took 30 hours to upload using rsync in one single run).

Still, it only has to be done once; on next runs, only new or changed tunes get scanned, and only new or changed fids are uploaded by rsync.

If you have a huge amount of tunes on your player, adding a single album using mp3tofid+rsync might take less time than it takes emplode+player just to rebuild the database.

But speed was not really the most important issue for me to write mp3tofid. I just did not want to keep track of all changes and apply these changes to my player manually.

I keep changing things to my mp3 collection, like
- changing id3 tags
- re-ripping badly ripped cd's
- re-encoding using better encoders or encoding options
- renaming tunes and directories
- deleting tunes and directories.
- adding tunes, of course
Whatever I do, mp3tofid and rsync will see the changes and apply them to the player automatically.

If you are going to upload a huge amount of tunes a first time using rsync, you may consider to mke2fs your fid filesystems before doing so (don't forget to save your config.ini file). Uploading to a freshly created, empty fs is much faster than overwriting old stuff. You will also get rid ot the huge fids directories that stay huge otherwise, even if you delete their contents.

Pim

Top
#120609 - 16/10/2002 23:49 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
ok, let me clarify my predicament. i noticed yesterday that one of my tags aren't capitalized right (Dj instead of DJ). i guess the tests doesn't check for capitalization changes.

but otherwise, i'm enjoying the mp3tofid/rsync otherwise. can't wait til ignore as child and "other" file types gets implemented. thanks for developing it.

Top
#120610 - 17/10/2002 08:00 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
i noticed yesterday that one of my tags aren't capitalized right (Dj instead of DJ). i guess the tests doesn't check for capitalization changes.

Well, the windows filesystem is case insensitive, even under cygwin. So if you only change capitalization, mp3tofid will still be able to open the file, and conclude it did not change. But if you edit the id3 tag inside the file as well, the modification date ought to change, and mp3tofid would notice. Perhaps your id3 tag editor resets the modification date after its change?

can't wait til ignore as child

I know what you mean. The other day, my player randomly played a 0db sine wave off my test cd ...

and "other" file types gets implemented

Which other types in particular? WMA is out, because of its closed nature. OGG and FLAC should be possible, I guess.

Pim

Top
#120611 - 17/10/2002 09:00 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
let me try this again. yeah, your right. mp3tagstudio doesnt modify the date. it works, just user error.

as for unsupported files, i need to sync up my wav files so i can do my test tones also. i just dont believe in mp3 test tones..

and how about this for a feature.... the ability to put a jemplode only soup file in a directory (probably in var), and let mp3tofid interpret it and convert to a playlist.

Top
#120612 - 17/10/2002 09:34 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
argh. check out what i did. i took off the "keep original date/time" toggle, and applied a tag renaming scheme to my whole collection. mp3tofid catches this, and rescans. there are many mp3s that stay the same, but just had the modified date changed.

when i try to rsync, it starts to do the whole collection again. i guess it only checks if theres a change in the modified date. i know this is not mp3tofid's fault, but isnt there a switch to make rsync make sure that nothing was changed?

Top
#120613 - 17/10/2002 10:07 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
isnt there a switch to make rsync make sure that nothing was changed?

You shouldn't need to worry; rsync will detect the files did not change; both ends are exchanging checksums. But both ends will need to read the whole file, so it will be slower. As a result, the fids on the player will get the modification time of the source fids, without actually transferring their contents.

Pim

Top
#120614 - 17/10/2002 10:14 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
as for unsupported files, i need to sync up my wav files

As wav files do not contain tags, I would have to add code that takes the title from the filename. This would be useful for mp3 too, as this is what emplode does now if a title tag is not found. I'll put that on my todo list.

the ability to put a jemplode only soup file in a directory

Uhm, now's the time to shamefully admit I have never tried to use jempeg ...

Pim

Top
#120615 - 18/10/2002 16:40 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
question.. i'm going to start using a preinit script to run rsync on startup, so i need to know if its safe to run it while the player app is still up? and if it eventually gives up if it doesnt find a network connection (i.e. in the car).

is there any sure way to know that i even have a network connection? somewhere in /proc maybe? so i can make the script conditional.

also, even though the database and playlist is functional, everytime i run emplode after rsync is done, it has to add/change something (litte red cross). i was thinking of doing a file compare, so you can implement what happens to the database/playlist generation code. or you can do it yourself.

Top
#120616 - 19/10/2002 10:44 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
ok, another feature request. i have different size hard drives... and i currently use the -2 50 switch. i need a celing on the smaller drive so i dont go over capacity. but until the time comes, i want to split up the mp3s evenly.

Top
#120617 - 19/10/2002 13:54 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
dmuench
stranger

Registered: 15/01/2002
Posts: 37
Loc: Honeoye Falls, NY
I just finished rsyncing the new fids and all to my empeg, and it works great now. No problems at all. I love the program, thanks so much for writing it.

Top
#120618 - 20/10/2002 11:57 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
is there any sure way to know that i even have a network connection?

If you get your IP from a DHCP server, you can see with ifconfig that you did not get an IP number.Get ifconfig from a Debian image.

somewhere in /proc maybe? so i can make the script conditional.

you can see in /proc/empeg_power whether you're on AC power or on car power.

even though the database and playlist is functional, everytime i run emplode after rsync is done, it has to add/change something

I noticed that too. I think it's because of some timestamp, not because of something in the database. I tested this in an earlier mp3tofid version; the databases created by emplode and by mp3tofid were 100% identical. With current versions they are not, because mp3tofid is leaving some tags out of the database that do not have to be there. This makes the database, and thus the player's memory usage, significantly smaller.

Pim

Top
#120619 - 20/10/2002 12:10 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
I have different size hard drives... and i currently use the -2 50 switch. i need a celing on the smaller drive so i dont go over capacity

In order to prevent unnecessary upload, it has to be predictable for any single file on which drive it will end. So I can't dynamically put the the file on the drive with the most diskspace like emplode does.

If it turns out one drive is getting full, then just change the percentage (ie "-2 51" or "-2 49"). You can check the diskspace on your host pc using du -L -s .../empegfids/drive?

I guess this makes it hard to fill both drives to the last byte, but you can't have everything in life ...

Pim

Top
#120620 - 20/10/2002 17:07 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
i'm going to start using a preinit script to run rsync on startup, so i need to know if its safe to run it while the player app is still up

I guess it would be safe to run rsync as a daemon while the player app is running, and drives are mounted read-only.

I would guess it is not safe to write to fids and databases while the player app reads them.

Pim

Top
#120621 - 21/10/2002 00:01 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
i have found that running rsync while the player is running is incredibly slow even while the player's on standby. i'm guessing the player app is running at a higher cpu priority and all memory is filled with song data.

which leads me to ask... is there anyway aside from hitting q in terminal mode to kill the player? i've tried to just use kill -9 pid but it always restarts.

Top
#120622 - 21/10/2002 11:45 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
is there anyway aside from hitting q in terminal mode to kill the player?

I wouldn't know. Perhaps it is best to ask again outside the context of this thread, and get a wider audience. I'd be interested too...

Pim

Top
#120623 - 21/10/2002 11:57 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
Daria
carpal tunnel

Registered: 24/01/2002
Posts: 3937
Loc: Providence, RI
It's evil, but you can:
-move the binary out of the way
-kill "player"
-init has nothing to respawn
-don't forget to move it back or you will be sad later

Top
#120624 - 21/10/2002 12:57 Re: mp3tofid 3.00 - now for Windows too [Re: image]
jaharkes
enthusiast

Registered: 20/08/2002
Posts: 340
Loc: Pittsburgh, PA
You could use kill -STOP <playerpid> which should freeze the player. Run rsync and then kill -9 <playerpid>, and the player will restart and reload the new databases. Although the player is a threaded process, so I'm not sure whether you can just kill -STOP the lowest player pid, or if you have to stop all of them individually.
_________________________
40GB - serial #40104051 gpsapp

Top
#120625 - 21/10/2002 18:15 Re: mp3tofid 3.00 - now for Windows too [Re: dmuench]
dmuench
stranger

Registered: 15/01/2002
Posts: 37
Loc: Honeoye Falls, NY
I guess I spoke too soon.

I just added a new album and re-ran mp3tofid, and it acted as if I had changed all the files - rescanned every file, and generated a new fid for every song. I was suspicious, so I ran rsync with the dry run option, and sure enough, it wanted to delete everything on the player and start over again.

I am sure this is something to do with me using XFS on my server, but I doubt that XFS randomly reassigns inode numbers whenever it feels like it. I am wondering if I am overflowing anything with the high inode numbers (the latest songs I just added all have inodes in the 373 million range).

Top
#120626 - 23/10/2002 03:33 Re: mp3tofid 3.00 - now for Windows too [Re: dmuench]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
Can you reproduce this, using a subset of your collection and a test fid dir?

I doubt that XFS randomly reassigns inode numbers whenever it feels like it

It's unlikely, but not impossible. This is easily detected. Just type ls -li in an mp3 dir, reboot and look again. smbfs on Linux turned out to give different results on each boot.

I am wondering if I am overflowing anything with the high inode numbers (the latest songs I just added all have inodes in the 373 million range)

There's nothing that can overflow anymore (at least not sooner than with other tools). A tune is now uniquely identified by a string built by the major and minor number of the device holding the filesystem containing the tune and the inode number of the tune. This string is used as a key to look up the fid number in a gdbm database. using -d dimMsS debug options you can follow all gdbm activity.

In theory, a 64-bit inode number (yours still look like 32-bit) could overflow the sprintf() function, but then still it would come up with a unique string.

So I can think of a number of explanations for this behaviour:
- your filesystem does not have persistent inode numbers
- you somehow lost the mp3tofid database
- you used the -n switch to force generating a new mp3tofid database
- the major/minor number of the filesystem changed caused by fdisk or lvm operations
- a weird bug

If you did lose the mp3tofid database, by accident or by using the -n switch, try restoring it from the player. This is why it is backed up there. Restore it, delete the fids on your host PC and run mp3tofid again. mp3tofid will have to scan everything again, but your fid numbers will be back to what they were.

Pim

Top
#120627 - 23/10/2002 14:59 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
dmuench
stranger

Registered: 15/01/2002
Posts: 37
Loc: Honeoye Falls, NY
I'll do some testing. I didn't delete the database or use -n though. And I am using EVMS, but one of it's features is persistent major/minors. I'll let you know what I find out.

Top
#120628 - 23/10/2002 15:14 Re: mp3tofid 3.00 - now for Windows too [Re: dmuench]
dmuench
stranger

Registered: 15/01/2002
Posts: 37
Loc: Honeoye Falls, NY
A lightbulb just went off in my head. Permit me to place my foot in my mouth.

I still have to test, but I am sure it's the major/minor issue. What I said is true, one of the features of an EVMS native volume is persistent major/minors. However, the volume that my mp3s are on is not a native, but a compatibility volume (dynamic major/minors) because I haven't had time to convert it yet. And I am pretty sure I did something to cause it's minor number to shift between mp3tofid runs.

So that must be the cause. Can I suggest to add a very visible section to the README about this, and I'd mention LVM volumes and EVMS compatibility volumes specifically. EVMS native volumes won't have this problem.

Thanks..

Top
#120629 - 24/10/2002 07:22 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
here's what i've discovered about the windows version. if i modify a file using winamp (or anything else for that manner), mp3tofid immediately recognizes it as a new file (i see 1 empty fids in the summary screen), and deletes the old fid. not sure why.

but the weird thing is... rsync says it only wrote < 10000 bytes, but took 10 seconds to complete after the initial scan. i dont understand whats happening in rsync i guess.

Top
#120630 - 24/10/2002 07:41 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
here's what i've discovered about the windows version. if i modify a file using winamp (or anything else for that manner), mp3tofid immediately recognizes it as a new file (i see 1 empty fids in the summary screen), and deletes the old fid. not sure why.

This all depends on how your id3 tag editor works. If it writes out a new file, deletes the old one and renames the new file to the old name, then you have created a new file with a new inode number. If it just opens the file in read/write mode and makes the changes inside the file, then the inode number remains the same.

i dont understand whats happening in rsync i guess

You can increase rsync's verbosity by placing the --verbose on the command line multiple times.

Pim


Top
#120631 - 24/10/2002 09:50 Re: mp3tofid 3.00 - now for Windows too [Re: dmuench]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
Thanks for your suggestion. I added the major/minor stuff in the lookup key as to enable support for cross-filesystem symlinks and mountpoints within your mp3 tree.

It did not occur to me that major/minor numbers could change at all until I read your report.

Pim

Top
#120632 - 09/11/2002 11:22 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
i've been noticing that some mp3s that i have in the collection shows as invalid format when played. i decided to erase and re-upload using emplode, and it worked fine. could a bad parse of the tag cause this?

also, have you noticed that when the rsync algorithm is used, it takes longer than just doing the whole file. I guess its just the weak processor of the empeg making the checksums thats slowing it down. If i edit an id3v2 tag, I get around 350k/sec regular, and 700k w/ the --whole-file switch.

Top
#120633 - 10/11/2002 19:54 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
If i edit an id3v2 tag, I get around 350k/sec regular, and 700k w/ the --whole-file switch

The problem with id3v2 tags is that they are at the beginning of the file, rather than at the end, like id3v1 tags. If you insert a id3v2 tag, or when its size increases, the mp3 frames are shifted to a different location in the file. As far as I can tell, rsync cannot detect this and compares the whole file, only to find out it has to transmit everything again, because none of the file parts are identical.

Pim

Top
#120634 - 11/11/2002 11:12 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
tms13
old hand

Registered: 30/07/2001
Posts: 1115
Loc: Lochcarron and Edinburgh
In reply to:

If you insert a id3v2 tag, or when its size increases, the mp3 frames are shifted to a different location in the file. As far as I can tell, rsync cannot detect this and compares the whole file, only to find out it has to transmit everything again, because none of the file parts are identical.


If that happens, your rsync isn't working correctly. The rsync protocol is designed to find common parts of files so that it can cope with this kind of thing - otherwise it would also fail on e.g. tar files when a file is added near the beginning. (Of course, compressed tar files are a different matter...)
_________________________
Toby Speight
030103016 (80GB Mk2a, blue)
030102806 (0GB Mk2a, blue)

Top
#120635 - 11/11/2002 12:35 Re: mp3tofid 3.00 - now for Windows too [Re: image]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
so anyway, regarding the fact that some files are being considered invalid format... here are the two tag files that were generated for the same file... one by emplode and one for mp3tofid.

emplode
artist=New Energy
bitrate=fs192
codec=mp3
ctime=1037042767
duration=410540
length=9895462
offset=42492
samplerate=44100
title=Energetic Wave (Nu-Nrg Original Mix)
type=tune
year=2002

mp3tofid
type=tune
artist=New Energy
bitrate=fs192
codec=mp3
ctime=1034559218
duration=410514
frames=15715
length=9895462
loadfrom=/cygdrive/e/mp3/2002/1002/new_energy-energetic_wave_nu-nrg_original_mix.mp3
offset=1540
samplerate=44100
title=Energetic Wave (Nu-Nrg Original Mix)
year=2002

parsing error in mp3tofid?

Top
#120636 - 11/11/2002 13:24 Re: mp3tofid 3.00 - now for Windows too [Re: tms13]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
You're right, I should have just tried it instead of trying to understand the sources.

Pim

Top
#120637 - 11/11/2002 14:31 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
here's an anomoly. try to just change the datestamp of one of the mp3s. when you rsync, it still appears to transmit the whole file. nothing changed at all, except the stamp. i'm going to do a --verbose 4 times and capture the log. hopefully, you can understand whats happening.

Top
#120638 - 11/11/2002 15:42 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
offset=42492

That's it. mp3tofid assumes it can find the start of the first mp3 frame in the first chunk it reads.
The chunksize is 40960 (#define INPUT_BUFFER_SIZE (5*8192)) in scanmp3.c.

Your mp3 seems to have a much larger id3v2 tag than I then the ones I tested with.

The lazy workaround to this bug is to increase the buffersize and recompile.

Unfortunately I will not be able to help, as I am packing for a three week trip to Asia.

Pim

Top
#120639 - 11/11/2002 17:39 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
If timestamp and/or size are different on each side, rsync has to check checksums of each side. This might be very slow if the empeg is involved, like you have seen in the case of editing id3 tags.

Pim

Top
#120640 - 12/11/2002 00:14 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
but the point is, emplode saw the offset at 1540. its the same track so... in theory, shouldn't all the parameters found be the same in both tag files?

edit: whoops. i'm dumb. it was mp3tofid that saw the offset at 1540. i have found that the files that were failing were because of un-standard hidden id3v2 tags, which i promptly erased. everything is fine now.


Edited by iMaGe (12/11/2002 01:01)

Top
#120641 - 12/11/2002 04:58 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
It's a bug nonetheless. I'll try to fix it when I return.

The other notable differences are seen in ctime and duration. The ctimes are inherently different, as mp3tofid uses the ctime of the source file, while emplode uses the ctime of the fid file on the player. The calculated duration is different too, but then I have yet to see two programs that come up with the same duration in milliseconds. I could be wrong, and libmad could be wrong, but emplode might as well be wrong. Then again, the player copes well with wrong duration tags, as I have seen during development, when mp3tofid durations were way off.

But beware of different offset or length. They have to match exactly.

Pim

Top
#120642 - 12/11/2002 08:06 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
i've found a bug in your compile of rsync. it seems to have the rolling checksum broken. with the --stats switch, if you only transfer one file, Matched data:will always be the same size as the block-size. even if there is no difference in the file. not really sure if its just a display error or not.

Top
#120643 - 12/11/2002 10:01 Re: mp3tofid 3.00 - now for Windows too [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
You could try the debian version (2.3.2) to see whether there's any difference.
My experience with the debian version is that it locks up when transferring/comparing huge numbers of tunes (>14000 in my case).

Of course, the problem you are seeing might also be caused by cygwin's rsync.
For testing, it would be nice to have a replacement for that too.

I'd be interested to see what you have found out. Now's the time to shut down my computers and leave for the airport.

Pim

Top
#120644 - 12/11/2002 10:54 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
dmuench
stranger

Registered: 15/01/2002
Posts: 37
Loc: Honeoye Falls, NY
It's been working great since. I've got a feature suggestion for when you get back from your trip though - I've got some music (holiday music) that I really don't want on my empeg. Can I suggest either a command line switch to exclude a directory(s) from mp3tofid scanning, or maybe a flag file we could place in a directory that would signal mp3tofid to ignore the directory (Maybe ".no_empeg" or something)? Either would be nice, I can see plusses and minuses for either solution. Thanks..

Top
#120645 - 24/11/2002 11:09 Re: mp3tofid 3.00 - now for Windows too [Re: pim]
smu
old hand

Registered: 30/07/2000
Posts: 879
Loc: Germany (Ruhrgebiet)
Hi.

Couldn't you use an md5 hash over the mountpoint _name_ instead of the partitions major/minor? By that, mp3tofid wouldn't recreate the whole database if the drive and/or partition changes its position, which can happen quite often on some systems (like mine: I often add/remove harddiscs or controllers, but the mount point for partitions always stays the same because I mount them by volume labels).

cu,
sven
_________________________
proud owner of MkII 40GB & MkIIa 60GB both lit by God and HiJacked by Lord

Top
#120646 - 04/12/2002 19:02 rsync > 2.3.2 broken [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
i've found a bug in your compile of rsync. it seems to have the rolling checksum broken.

Yes, it appears rsync 2.5.5 on the empeg will always transfer a file, even if only the timestamp is different. This is a bug that supposedly was in 2.5.4 but has been fixed in 2.5.5. But the problem also appears to be there in version 2.4.6, which can be found here. It's also there in the latest CVS version.

The only version that does not appear to have this bug is the 2.3.2 debian version. Unfortunately, this version has other problems.

I did all my testing against a linux x86 box with rsync 2.5.5.

I'm not sure what causes this bug. Maybe an ARM alignment problem, or a gcc bug. I can't find anything in the rsync mailing archives either.

Pim

Top
#120647 - 04/12/2002 19:25 Re: mp3tofid 3.00 - now for Windows too [Re: smu]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
Couldn't you use an md5 hash over the mountpoint _name_ instead of the partitions major/minor?

I don't think there's a fast and portable way to discover the mountpoint name. major and minor numbers are easy, though, as they are returned in the stat() data.

Thinking about it, I might just as well have ommitted the mountpoint data, unless it is different from the mountpoint data of the root of the mp3 tree. This would solve almost anyones problems, and it would still work in the rare cases where mountpoints exist within the mp3 tree.

Pim

Top
#120648 - 04/12/2002 23:13 Re: rsync > 2.3.2 broken [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
welcome back from asia btw. how was the trip?

as for rsync... i'm content w/ 2.3.2. don't really see a problem with it, and i was able to speed up the upload of a slightly modified mp3 file (id3 tag mods) by putting the checksum block size to 100000 bytes. it minimizes the transmittion of checksums/packets.

i was wondering if it would be possible to "import" a current fid/file structure on the empeg. there is a bug that got me by surprise. mp3tofid segfaults when you change operating systems (xp to 2k in my case). i deleted everything EXCEPT the mp3tofid association file, and still segfaulted. finally gave up, deleted the mp3tofid file, and started anew. but my point is... if you took all the tag files (______1) from the empeg, couldnt you parse them and rebuild the associations, depending on the fact that the import_location field is filled (mp3tofid and jemplode use that field and emplode seems to have omitted it). the only limitation is in windows based systems that used jemlode, you'd have to convert "c:\" to "/cygdrive/c".

speaking of jemplode, i'd really like to be able to have the hash value of an mp3 calculated and put into the tag file just like jemplode. i know that you don't use jemplode... but i use it to do the stuff that mp3tofid can't do (yet).

Top
#120649 - 05/12/2002 08:29 Re: rsync > 2.3.2 broken [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
welcome back from asia btw. how was the trip?

Thanks, I had a great time visting Bangkok, Hong Kong, Macau and Guangzhou.

as for rsync... i'm content w/ 2.3.2.

Lucky you. Mine kept locking up. Not during large fresh transfers, but during small changes inside 70 gigabyte of MP3's.

i was able to speed up the upload of a slightly modified mp3 file (id3 tag mods) by putting the checksum block size to 100000 bytes. it minimizes the transmittion of checksums/packets

You'll be transferring at least one 100.000 byte block then. Did you time different sizes as well?

i was wondering if it would be possible to "import" a current fid/file structure on the empeg

That's going to be hard, as plain empeg fids do not contain the "loadfrom" tag.

mp3tofid segfaults when you change operating systems (xp to 2k in my case)

mp3tofid should not segfault, but you lose anyway. mp3tofid since version 3 uses not only the inode number, but also the major and minor number of the filesystem to uniquely identify a file. Doing so, mp3tofid will handle the rare case of mountpoints within the mp3 tree, which could allow for duplicate inode numbers.

If you copy your mp3's to a different computer, every mp3 file will probably have different inode numbers and the filesystem containing them will probably have different major/minor device numbers. So all fid numbers will be invalid, and will have to be recalculated. And new fid numbers means yet again a new whole rsync transfer. Sorry, but this is how it's designed.

Pim


Top
#120650 - 05/12/2002 08:48 Re: mp3tofid with jemplode hashes [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
i'd really like to be able to have the hash value of an mp3 calculated and put into the tag file just like jemplode

Is that a plain md5sum hash? Should be doable, as mp3tofid reads the whole file anyway.

Pim

Top
#120651 - 05/12/2002 09:29 Re: rsync > 2.3.2 broken [Re: pim]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA

You'll be transferring at least one 100.000 byte block then. Did you time different sizes as well?


i tried a 500,000 block, but it seemed to take longer. didn't really want to to test it too much. i'll try it tonight.


That's going to be hard, as plain empeg fids do not contain the "loadfrom" tag.

If you copy your mp3's to a different computer, every mp3 file will probably have different inode numbers and the filesystem containing them will probably have different major/minor device numbers


i know that the plain empeg doesn't use the tag. but the tags already generated have them, as well as jemplode. to recover from an inode change, can't you parse the existing tags, find out the new inode/major/minor#, and repopulate the mp3tofid file, assuming that the file structure was the same.... (or just mass replace the path to wherever the files are).

Is that a plain md5sum hash? Should be doable, as mp3tofid reads the whole file anyway.

its a crc32 check on only the mp3. exclude the tags.

Top
#120652 - 04/02/2003 14:52 Re: stopping the player from the shell/hijack [Re: image]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
is there anyway aside from hitting q in terminal mode to kill the player? i've tried to just use kill -9 pid but it always restarts.

You can gracefully stop the player with "kill -INT <player-pid>"
and restart it with "kill -HUP <bash-pid>". Or use killall from
debian to automatically find the pid.

Another way to stop and start the player is to use hijack:
http://player-ip/?NODATA&SERIAL=q will stop the player,
http://player-ip/?NODATA&SERIAL=exit will start it.

You would need wget or something to do this from the player,
but there might be a more direct way to have hijack emulate
serial input. Anyone?

Pim

Top
#120653 - 04/02/2003 15:03 Re: stopping the player from the shell/hijack [Re: pim]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Yeah. You should be able to:

echo "q" > /proc/empeg_notify
_________________________
Bitt Faulk

Top
#120654 - 04/02/2003 16:26 Re: stopping the player from the shell/hijack [Re: wfaulk]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
echo "q" > /proc/empeg_notify]

Hmm... Does that work for you? Not for me, running
hijack v310. Would be cool, though.

Pim

EDIT: Oops, forgot to add


[output]
notify=1



to config.ini


Edited by pim (04/02/2003 16:34)

Top
#120655 - 04/02/2003 16:44 Re: stopping the player from the shell/hijack [Re: pim]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
I didn't know that was a requirement. It's how empegVNC sends button presses to the player.
_________________________
Bitt Faulk

Top
#120656 - 05/02/2003 14:51 Re: stopping the player from the shell/hijack [Re: wfaulk]
pim
addict

Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
Well, it did not work, I made a post about it and then read the FAQ.

Wrong order ...

But now it appears not to work even with the mentioned addition
to config.ini. Did you try this? Of couse, you need to install emptelnetd
to try.

Pim

Top
#120657 - 05/02/2003 14:57 Re: stopping the player from the shell/hijack [Re: pim]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Okay, no, I didn't try it.

All I know is that it works for button presses, et al. I'd try it here, but I can't attach my empeg to the network at work.
_________________________
Bitt Faulk

Top
Page 1 of 2 1 2 >