Hijack v310: support for new fids subdir structure

Hijack v310 is now released.

New with this version is automatic detection/support in khttpd playlist browsing for the new-style FIDS directory layout in v2beta13 and higher. The mp3tofid tool knows how to create this layout (bug free?), and once created, it is all transparent to Emplode, JEmplode, and emptool.

Basically, the old way that most of our players use is to stuff all of the tunes, playlists and tag files into one huge directory on each drive:

/driveX/fids/

But having massive numbers of files all in one place slows down the Linux kernel whenever it performs directory operations on them, so Roger and pals added support to v2beta13 and beyond for a more distributed directory structure.

If you think of a FID as a 32-bit number, the filename on disk is normally just that value as an 8-digit hex number, with the leading zeros suppressed. Under the new layout, the first five digits are used as a subdirectory name, and only the final three digits are used for the actual filename.

So, old way: /drive0/fids/12345
new way: /drive0/fids/_00012/345

Both ways now work with the playlist browser in khttpd in Hijack.

-ml

In reply to:

---

If you think of a FID as a 32-bit number, the filename on disk is normally just that value as an 8-digit hex number, with the leading zeros suppressed. Under the new layout, the first five digits are used as a subdirectory name, and only the final three digits are used for the actual filename.

So, old way: /drive0/fids/12345
new way: /drive0/fids/_00012/345

Both ways now work with the playlist browser in khttpd in Hijack.

---

Curious, how big would your playlist/amount-of-files have to be before this layered FIDs to appear faster? (When does the number of entries get to noticibly slow down the kernel?)

Me.

i seem to recall someone making a script that changed the old structure to the new by mv-ing everrything appropriately. my quick searching yielded no results. wondering if anyone has a copy of it to post?

Is this why visuals will occasionally sttuter? I thought it was because I was running out of hard drive space, then I deleted 10 gigs and didnt notice a difference.

Over in the General forum I posted a link to it. I just did a search for "shell fids" or "script fids" or something and limit it to the programming forum.

Is this why visuals will occasionally sttuter?

Unlikely. The only real benefit that the new directory layout gives is in database rebuilds, and we've not actually got around to measuring it, so it might not be much.

A large number of FIDs will cause directory indexes to be slower, but since the player has (usually) already pre-cached parts of the next few tracks asynchronously, a stutter would be unlikely to result.

Also, if this was causing the visuals to stutter, the music playback would stutter also. This could happen when memory gets low, which can happen if you've got a lot of tracks -- the database and the cache share the same space.

More likely is that you're (very occasionally) borderline on CPU usage, and the visuals are getting starved.

The answer to the "how big" question is that it is CPU and memory dependent. On the Empeg/RioCar player, the new layout probably won't save many cycles unless there are a couple thousand tunes/playlists or more -- each tune/playlist results in two files in the fids directory.

Most of us have more than a couple thousand fids by now.

IT shouldn't make much/any difference on a stock player, but it helps when using the web interface, and frees up some CPU for third-party apps that may be active simultaneously with the music player.

EDIT: And of course it speeds up FTP and mirrordir quite a bit under some conditions, and should speed up database rebuilds by a measurable amount.

Cheers

Curious, how big would your playlist/amount-of-files have to be before this layered FIDs to appear faster?

Like has been said before, this greatly depends on the number of files in your fids dir. On my player, this number was approaching 15.000 per drive.

Changing this to the new structure made a huge difference in responsiveness to shell commands. Especially "rwm" took seconds instead of minutes, and rsync started transferring much sooner. I was so enthusiastic that I made the new scheme the default for mp3tofid.

One important thing is to create new fids directories after converting, as the old fids directories stay big, even if you empty them; ie something like

mkdir fids.new

mv fids/* fids.new

rmdir fids

mv fids.new fids

Even better would be to start with freshly mke2fs'ed filesystems.

Pim

as the old fids directories stay big, even if you empty them

To clarify what Pim's saying: As you add files to a directory, the amount of space used by that directory gets larger -- it allocates more space to list those files in.

With the ext2 (and other) filesystem, when you delete files from a directory, the space allocated to the directory is never (or is it just rarely?) reclaimed.

So, if you have a directory that once had lots of files in it, and you want to shrink it, you need to move the contents somewhere safe, and then delete and recreate it, like Pim's saying.

And 2.0b13 (and associated Emplode) see this transparently? i.e. If I run this update schript (<- boston-ese) they won't notice or give a d@mn?
(And I've been familiar with the whhole slowdown thing; just never knew /why/. Now it makes sense.)

Me.

I dunno which script you are looking at, but if you cause the /fids/ stuff to be partially or completely moved to the new subdirectory scheme, emplode et al will continue to work in v2beta13. Over time, as changes are made with emplode, files will again be dumped into the single fids directory, so after any major syncs you might want to re-shuffle things into the subdirs again. Just to keep it tidy -- it will continue working even if you don't do this.

-ml

Heh. It made the swear word an e-mail link. Cute.

Anyways, here's the shell schript, I THINK.. Obviously, modify it if you only have one drive.

That shell script is NOT CORRECT!!! DO NOT USE IT!!!

-ml

EDIT: it doesn't move all of the files, and it doesn't use the correct new directory naming scheme, mostly since that particular script pre-dates general knowledge here of how the real scheme got implemented.

-ml

Eerp, sorry. Shoul;da looked at it closer before posting. That's the only semi-script-looking thing that the search came up with.. Gotta go search for the right one, if it was ever posted.
(Hell, I'm thinking it'd actually be easier (for me; I'm not big on scripting; go figure) and faster to just write a small C program to handle it all. Anyone interested? [I'm learning C. Gotta put it to good use. Heh.])

Me.

Sorry for waiting so long to reply, but thanks for the new version. I am on version 312 right now, and jEmplode now downloads all of the mp3s to my computer instead of leaving 0KB files (those are the ones uploaded with mp3tofid). So if anyone was having this problem, it seems to be fixed.

That shell script is NOT CORRECT!!! DO NOT USE IT!!!

So does anybody have a script or program that will work handy? I was considering converting the structure on mine to see if it noticably speeds up the painfully long database rebuilds.

-Mike