#139561 - 03/02/2003 14:28
Hijack v310: support for new fids subdir structure
|
mlord
carpal tunnel
Registered: 29/08/2000
Posts: 10005
Loc: Canada
|
Hijack v310 is now released.
New with this version is automatic detection/support in khttpd playlist browsing for the new-style FIDS directory layout in v2beta13 and higher. The mp3tofid tool knows how to create this layout (bug free?), and once created, it is all transparent to Emplode, JEmplode, and emptool.
Basically, the old way that most of our players use is to stuff all of the tunes, playlists and tag files into one huge directory on each drive:
/driveX/fids/
But having massive numbers of files all in one place slows down the Linux kernel whenever it performs directory operations on them, so Roger and pals added support to v2beta13 and beyond for a more distributed directory structure.
If you think of a FID as a 32-bit number, the filename on disk is normally just that value as an 8-digit hex number, with the leading zeros suppressed. Under the new layout, the first five digits are used as a subdirectory name, and only the final three digits are used for the actual filename.
So, old way: /drive0/fids/12345
new way: /drive0/fids/_00012/345
Both ways now work with the playlist browser in khttpd in Hijack.
-ml
|
|
Top
|
|
|
|
#139562 - 03/02/2003 15:52
Re: Hijack v310: support for new fids subdir structure
[Re: mlord]
|
foxtrot_xray
addict
Registered: 03/03/2002
Posts: 687
Loc: Atlanta, Georgia
|
In reply to:
If you think of a FID as a 32-bit number, the filename on disk is normally just that value as an 8-digit hex number, with the leading zeros suppressed. Under the new layout, the first five digits are used as a subdirectory name, and only the final three digits are used for the actual filename.
So, old way: /drive0/fids/12345
new way: /drive0/fids/_00012/345
Both ways now work with the playlist browser in khttpd in Hijack.
Curious, how big would your playlist/amount-of-files have to be before this layered FIDs to appear faster? (When does the number of entries get to noticibly slow down the kernel?)
Me.
Mike 'Fox' Morrey
128BPM@124MPH. Love it!
2002 BRG Mini Cooper
|
|
Top
|
|
|
|
|
#139563 - 03/02/2003 16:20
Re: Hijack v310: support for new fids subdir structure
[Re: mlord]
|
image
old hand
Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
|
i seem to recall someone making a script that changed the old structure to the new by mv-ing everrything appropriately. my quick searching yielded no results. wondering if anyone has a copy of it to post?
|
|
Top
|
|
|
|
|
#139565 - 03/02/2003 16:56
Re: Hijack v310: support for new fids subdir structure
[Re: image]
|
tonyc
carpal tunnel
Registered: 27/06/1999
Posts: 6108
Loc: Pittsburgh, PA
|
Over in the General forum I posted a link to it. I just did a search for "shell fids" or "script fids" or something and limit it to the programming forum.
|
|
Top
|
|
|
|
|
#139566 - 03/02/2003 17:12
Re: Hijack v310: support for new fids subdir struc
[Re: eliceo]
|
Roger
carpal tunnel
Registered: 18/01/2000
Posts: 4572
Loc: London, UK
|
Is this why visuals will occasionally sttuter?
Unlikely. The only real benefit that the new directory layout gives is in database rebuilds, and we've not actually got around to measuring it, so it might not be much.
A large number of FIDs will cause directory indexes to be slower, but since the player has (usually) already pre-cached parts of the next few tracks asynchronously, a stutter would be unlikely to result.
Also, if this was causing the visuals to stutter, the music playback would stutter also. This could happen when memory gets low, which can happen if you've got a lot of tracks -- the database and the cache share the same space.
More likely is that you're (very occasionally) borderline on CPU usage, and the visuals are getting starved.
|
|
Top
|
|
|
|
|
#139568 - 03/02/2003 17:36
Re: Hijack v310: support for new fids subdir structure
[Re: foxtrot_xray]
|
pim
addict
Registered: 14/11/2000
Posts: 428
Loc: The Hague, the Netherlands
|
Curious, how big would your playlist/amount-of-files have to be before this layered FIDs to appear faster?
Like has been said before, this greatly depends on the number of files in your fids dir. On my player, this number was approaching 15.000 per drive.
Changing this to the new structure made a huge difference in responsiveness to shell commands. Especially "rwm" took seconds instead of minutes, and rsync started transferring much sooner. I was so enthusiastic that I made the new scheme the default for mp3tofid.
One important thing is to create new fids directories after converting, as the old fids directories stay big, even if you empty them; ie something like
mkdir fids.new
mv fids/* fids.new
rmdir fids
mv fids.new fids
Even better would be to start with freshly mke2fs'ed filesystems.
Pim
|
|
Top
|
|
|
|
|
#139569 - 03/02/2003 17:50
Re: Hijack v310: support for new fids subdir structure
[Re: pim]
|
Roger
carpal tunnel
Registered: 18/01/2000
Posts: 4572
Loc: London, UK
|
as the old fids directories stay big, even if you empty them
To clarify what Pim's saying: As you add files to a directory, the amount of space used by that directory gets larger -- it allocates more space to list those files in.
With the ext2 (and other) filesystem, when you delete files from a directory, the space allocated to the directory is never (or is it just rarely?) reclaimed.
So, if you have a directory that once had lots of files in it, and you want to shrink it, you need to move the contents somewhere safe, and then delete and recreate it, like Pim's saying.
|
|
Top
|
|
|
|
|
#139575 - 09/02/2003 13:15
Re: Hijack v310: support for new fids subdir structure
[Re: mlord]
|
MP3944
journeyman
Registered: 25/05/2002
Posts: 55
Loc: Grove City, OH, USA
|
Sorry for waiting so long to reply, but thanks for the new version. I am on version 312 right now, and jEmplode now downloads all of the mp3s to my computer instead of leaving 0KB files (those are the ones uploaded with mp3tofid). So if anyone was having this problem, it seems to be fixed.
|
|
Top
|
|
|
|
|
#139576 - 12/02/2003 18:15
Re: Hijack v310: support for new fids subdir struc
[Re: mlord]
|
mcomb
pooh-bah
Registered: 31/08/1999
Posts: 1646
Loc: San Carlos, CA
|
That shell script is NOT CORRECT!!! DO NOT USE IT!!!
So does anybody have a script or program that will work handy? I was considering converting the structure on mine to see if it noticably speeds up the painfully long database rebuilds.
-Mike
|
|
Top
|
|
|
|
|
|
