fidsift.sh -- rearrange /drive?/fids/ into subdirs

Posted by: mlord

fidsift.sh -- rearrange /drive?/fids/ into subdirs - 07/10/2003 17:12

Quite some time ago, the player software was enhanced to be able to deal with /fids/ directories that are broken up into smaller subdirs. This can speed up database rebuilds and filesystem checks etc, and is a worthwhile thing to have.

The Emplode and emptool software still just upload everything into /drive?/fids/, but the player software (and Hijack) can handle things in either format, including a mix of the two methods.

So.. here's my small script for (re)sifting the tune and tag files into appropriate subdirs, on a one or two drive RioCar/Empeg player.

To use, upload via FTP to /fidsort.sh, and set the exec permissions:

ftp myplayer
site rw
cd /
put fidsift.sh
chmod 0755 fidsift.sh
site ro
quit
Then connect via the serial port, hit control^C to gain control of the player, and do this command: /fidsift.sh

Then go and have coffee or cola, and when you get back it will eventually finish, and you can view the results in /drive0/fids/ and /drive1/fids/. As new tunes are later uploaded, unsifted files will acculumate again, and you might want to periodically rerun the script in the same way as the first time, to sift the newly uploaded tunes into their subdirs (not necessary, but nice).

The script is attached -- EDIT: there's a better version later in this thread..

Cheers

Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 07/10/2003 19:24

Want me to do a "before and after" timing of database rebuild? ~22,000 tracks I think.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 19:41

Now that could be very educational!

I'm also wondering if the script will just crap out with that many fids.. go ahead and give it a whirl!

(it won't actually damage anything if it does crap out.. maybe I'll try a test here..)
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 19:49

I trust that if it craps, only part of the fids will be moved and emplode will just handle it?
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 19:51

Actually, the place where it might crap would just result in nothing happening.

And if it did only do it partially, no big deal -- emplode doesn't know nor care, and neither does the player or hijack. Everything still just works
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 19:52

Hang on a few minutes.. I'm generating a dummy fids directory with 22000 "tracks"..
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 19:56

I just tried it (on my laptop) with 25000 fids (over 50000 files). No problem. Not tried on the player (would take me too long), but it's safe.

Cheers
Posted by: image

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 20:03

can hijack be modified to run this script automatically after uploading? like i would love it before the player restarts, so the player would still be expected to be in rw mode.
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 20:16

Seems like it might make (already long) sync's longer.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 20:16

Okay, here's a slightly improved version.

This one shrinks the /fids/ directory afterwards to improve lookups further, and also copes correctly with a /fids/ directory that contains no files.

Cheers
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 20:17

Faster updates than HiJack!
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 20:18

This script, after the initial run, is actually quite fast. So it wouldn't really slow down most syncs at all (not noticeably), unless uploading several hundred tracks.

I thought about building it into hijack, but a better option is just to bind it into the menu system as a user app.. Gotta write that new menu thingie Real Soon Now (the "site exec" stuff and friends).

Cheers
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 07/10/2003 20:26

chmod 0755 = what does this equate to?

I'm using WS_FTP and it doesn't do numbered chmods... only checkboxes.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 20:27

rwxr-xr-x == 0755

User(owner): read/write/execute
Group: read/execute
Other: read/execute
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 20:39

Thanks for the edit too.
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 20:52

Update: The FidSift is currently taking longer to run (at least this first time) than the "pre sift" sync... will it be a photo finish or is there a long way to go?
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 07/10/2003 21:05

The first sift will be slow. But you should be getting the odd progress indication in the form of a mkdir: created directory `_00000' style of message once in a while.

Cheers
Posted by: SE_Sport_Driver

Way to go Mark! - 07/10/2003 21:33

Okay, I have 20,919 mp3s on the player (I always forget how many I have because the player doesn't always removed duplicates when repeating the root playlist like it used to) with 6.26GB free of 130GB. Anyway... the process:

1. Installed latest Hijack.

2. Connected via emplode and added one 4.54mb file.

3. Pre "FidSift" Sync
Start: 10:11
End: 10:28
Elapsed: 0:17 (I was surprised it was this quick!)

4. Installed and ran FidSift.
Started sifting drive0: 10:52
Started sifting drive1: 11:11
Ended: 11:34
Elapsed: 0:42
Drive0 went up to about _0004d and Drive1 went to _0005b. As Mark pointed out, this shouldn't take this long after the initial run.

5. Connected via emplode, replaced same 4.54MB file.

6. Post "FidSift" Sync
Start: 11:35
End: 11:43
Elaspsed: 0:08

That's over a 100% speed increase!
Posted by: mlord

Re: Way to go Mark! - 07/10/2003 21:36

Ahh, good. I was hoping for some kind of measurable improvement there. I forgot to do the "before" timings on my own players, and I have not written a script to convert the files back to the old naming/layout scheme, so..

WooHoo!

(thanks for trying it out)
Posted by: SE_Sport_Driver

Re: Way to go Mark! - 07/10/2003 21:38

No, thank you! Database rebuilds were painfully slow but I can live with 8 minutes now! I assume that everyone can get rebuilds to be over twice as fast too.
Posted by: mlord

Re: Way to go Mark! - 07/10/2003 21:40

Mmm.. I suppose I could have the script do both drives in parallel, which would cut the time for the first sift down by quite a bit. But that's a one-time only event (per player), so no big deal. And I don't have a two-drive unit here to test with, so I think I'll just leave well enough alone!

A subsequent "sift" should be near instantaneous, unless a thousand or so (or more) tracks are modified with emplode/emptool.

Cheers
Posted by: loren

Re: Way to go Mark! - 07/10/2003 21:45

Sweeet. Nice job Mark. So can we expect this to be built into hijack soon? If so i think i'll wait. I'm going to be completely rebuilding my player soon anyhow. It would rock if this could be run after a certain number of tracks have been added. Could Jemplode do this somehow??
Posted by: mlord

Re: Way to go Mark! - 07/10/2003 21:49

Mmm.. Well, JEmplode over ethernet could use FTP for the file transfers, in which case it could just use the new subdir structure as it uploads tracks. Maybe Mike will add that someday.

I don't know about building it into Hijack -- so don't wait for that to happen. Once I someday implement the "run a command from the menu" feature it will be trivial to do. But I keep getting distracted here..

This is pretty simple to do as-is.. and even if it's only done once it will still make a difference. The only tricky bit is getting the script onto the player, but that only need be done the first time. From then on, it's still there and can be run just by hitting control^c and typing it's name (using the serial link). And it's only slow the first time!

Sleep time..

Cheers
Posted by: Foz

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 08/10/2003 00:47

Is this thing safe under the V3 alpha kernel?

-- Gary F.
Posted by: simspos

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 08/10/2003 01:18

Nice work Mark,.... people of the world (with obscenely large music collections) Rejoice!
Posted by: johnmcd3

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 08/10/2003 05:06

Yes.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into sub - 08/10/2003 06:17

The "v3alpha kernel" is the same as the v2-final kernel, so.. yes, it's safe.

It is also just fine with the v3alpha player software, too.

-ml
Posted by: mlord

Oh no! A serious flaw! - 08/10/2003 08:34

Okay, there's a flaw with this scheme.

When syncing with Emplode or emptool, the player stores new/modified FIDS in the top level /driveX/fids/ directory, rather than the subdirs. Same when doing deletes. Nothing wrong with that, other than some duplication.

BUT.. the player software does NOT search the top level directory if it finds a subdir.. OUCH. I was under the impression that it was smart enough to look in the right order, but it is actually even smarter than that -- smart enough not to "waste" the effort. This is a good thing, but it means that you MUST re-run fidsift.sh after each sync(Ugh!), and even that won't properly delete things.

Okay, I'll fix this, in the Hijack kernel, so that the player does "the right thing" whether it wants to or not. Later today, or tomorrow.

The plan is have the kernel automatically remap ALL /fids/ accesses to the subdirs when they (subdirs) exist. This way, syncs which affect existing fids will work correctly. New uploads can still be dropped into the top level /fids/ directories, if no appropriate subdir exists at the time. Not an issue for now.

Cheers
Posted by: image

Re: Oh no! A serious flaw! - 08/10/2003 09:56

my solution.

;@AC ;@EXEC /bin/fidsift.sh

cant use ;@EXEC_ONCE because that only executes after a reboot.

unless you uploaded over 500 files, the time diff is unnoticable.
Posted by: mlord

Re: Oh no! A serious flaw! - 08/10/2003 10:33

That will almost work. Requires a re-sync after the first sync, to ensure the database is rebuilt using the newly sifted files.. thus the problem.

Cheers
Posted by: mlord

Re: Oh no! A serious flaw! - 08/10/2003 12:20

Mmm.. now that I've traced through it some, it appears that the player IS doing the RIGHT THING(tm), at least some of the time. I need to play with it more to make sure.

At database rebuild time, it correctly looks for (eg.) /drive0/fids/100 before it looks for the same file in a subdirectory (eg.) /drive0/fids/_00000/100

So, as long as it remembers where it found each file for later (part of the database?), then it should be fine as is.

I'll check that now. Meanwhile, I have a hacked hijack version that automatically ensures that new uploads end up in the right places. Not released yet.

Cheers
Posted by: mlord

Re: Oh no! A serious flaw! - 08/10/2003 12:44

Yeah, it's fine.

My worries were for naught. All is well.

But I may still release the changes I made to Hijack, which ensure that the /fids/ directory stays sifted after the initial run.. But then again, a userspace tool can still do this equally well, so why bloat Hijack?

Cheers
Posted by: SE_Sport_Driver

Re: Oh no! A serious flaw! - 08/10/2003 15:54

I agree that there is no point in bloating HiJack. Could this user ap be executed via ethernet or ftp rather than serial?
Posted by: loren

Re: Oh no! A serious flaw! - 08/10/2003 16:35

Bloat hijack because it's one less thing we'll have to install and keep track of. And it'll be easier to run. Right? Wrong?
Posted by: Daria

Re: Oh no! A serious flaw! - 08/10/2003 19:50

Think of this as technical prose, and not really a good solution.

telnet empeg
rw
mv /empeg/bin/player /empeg/bin/player.disabled
cat > /empeg/bin/player << __EOF__
#!/bin/sh
echo "init will helpfully reboot if it can't run the player. Bad init. No biscuit."
return 0
__EOF__
chmod +x /empeg/bin/player
ps auxww|grep player|grep -v grep|awk '{print $2}'|xargs kill
sleep 3
mv /empeg/bin/player.disabled /empeg/bin/player
fidsift
kill -1 1
Posted by: peter

Re: Oh no! A serious flaw! - 09/10/2003 03:25

At database rebuild time, it correctly looks for (eg.) /drive0/fids/100 before it looks for the same file in a subdirectory (eg.) /drive0/fids/_00000/100

So, as long as it remembers where it found each file for later (part of the database?), then it should be fine as is.
It doesn't remember; it re-does the search every time.

Peter
Posted by: Roger

Re: Oh no! A serious flaw! - 09/10/2003 05:15

Yeah, but at database rebuild time, it does a directory-recursive search for FID files. At FID-lookup time, it has a specific strategy -- which might be the other way round, I don't recall.
Posted by: Shonky

Re: Oh no! A serious flaw! - 09/10/2003 05:44

OK Mark I just tried it on a 9436 song empeg - about 50Gb.

My force database rebuild consisted of changing the name of one playlist.

pre sift sync : 4:53
initial sift time was 18:27
post sift sync : 3:20

So not 100% like Brad but still faster. It obviously helps players with more files.

However, there was one problem! On my /drive1 it created a dodgy directory. This is a section of the inital sift...
Sifting /drive1/fids..

mkdir: created directory `_00000'
mkdir: created directory `_0'
mkdir: created directory `_00001'
mkdir: created directory `_00010'
mkdir: created directory `_1'
mkdir: created directory `_00011'
mkdir: created directory `_00012'
mkdir: created directory `_00013'
mkdir: created directory `_00014'
mkdir: created directory `_00015'
hijack: found new-style fids subdirectories
mkdir: created directory `_00016'
mkdir: created directory `_00017'
mkdir: created directory `_00018'


In the _0 directory is now a file called 10 which is the info type file. In the _1 directory is now a file 11 which is the matching mp3 file for the info file in the _0 directory.

According to a CSV export of my database (done after this song was upload I'm 100% certain) the fid of this track should have been 7172 decimal i.e. 0x1C04. So I have no idea what has happened and where it got possible 110 and 111 as the fids from. I don't understand regexps that well, so I can't see anything obviously wrong with your script.

I tried moving the files back to where they were and called them 1c040 and 1c041. I reran the sift program but this time it worked fine and put the files in the correct directory. So I have no idea what went wrong. I haven't listened to the particular song in awhile so it may not have been stored properly, although I didn't receive any sort of database errors via emplode.

Anyway thanks for doing it. It still sped up my rebuilds.
Posted by: mlord

Re: Oh no! A serious flaw! - 09/10/2003 05:57

Yeah, I think it is the other way around. Not a big deal, that, though it can be the source of some confusion. Database rebuilds look for non-subdir'd files first, then for the subdirectory versions. At playback time, the player looks into the subdirectories first, and then for the flat file versions. I think.

Rather than theoretically "fixing" that someday, I would very much prefer that a future release of the player software merely include code to automatically write new FIDS to the subdirectories when uploading. Unconditionally would be fine, but one could gate it by requiring that at least one subdir already exists or that the player be empty.

I have modified Hijack-v344 to automatically redirect ALL acceseses to the subdirs, player or otherwise, but only when performed through the symlinks in /empeg/fids?/ (not when done via /drive?/fids/, but only if at least one (any) subdir already exists. I didn't bother with the "empty player" case. To be released soon.

Cheers
Posted by: mlord

Re: Oh no! A serious flaw! - 09/10/2003 06:08

Mmm.. Shonky, that's rather odd -- and theoretically impossible from the way the script is written. I suppose maybe /bin/bash got low on memory or something. I will modify the script to turn on swap before/after running, to alleviate any such memory pressures.

The upcoming Hijack-v344 kernel includes code to automatically keep a sifted drive sifted after future sync operations. This code also replaces some existing code that was in the http server, so overall it doesn't contribute much bloat at all.

Cheers
Posted by: Daria

Re: Oh no! A serious flaw! - 09/10/2003 12:33

everything except the kill -1 1 at the end (which i hoped would make init respawn the player) worked. fidsifted without use of a serial port last night.
Posted by: SE_Sport_Driver

Re: Oh no! A serious flaw! - 23/11/2003 12:54

Bump. I thought I'd bump this thread for people interested in fidsift now that the new HiJack is coming.
Posted by: SE_Sport_Driver

help! fidsift - 19/09/2004 22:24

I decided to send fidsift.sh via TeraTerm (Hyperterminal type program) rather than ftp'ing since it would save me the extra step of hooking up an ethernet cable.

I did an "ro" and "rom" before changing to the root directory and using the "Send File" option.

Problem is, once I sent the file, it appears to have executed the file! Is this normal? It "looks" to be doing the right thing, but I didn't have a chance to be sure... I also never ran "chmod 0755 fidsift.sh".

Do I have anything to worry about? Just let it run? Why did it execute?

UPDATE: Looks like everything ran okay... but I see no trace of fidsift.sh being on the player... It opened the file but never saved it.. I've since ftp'd it to the root directory so it can be run in the future.
Posted by: SE_Sport_Driver

Re: Oh no! A serious flaw! - 19/09/2004 22:30

Could this line be added to config.ini to run fidsift from HiJack?

[hijack]
;@MENUEXEC fidsift /fidsift.sh
Posted by: wfaulk

Re: help! fidsift - 19/09/2004 23:15

You probably sent an ascii dump rather than transferring the file via xmodem or similar, and since a .sh file is just a series of commands to the shell being run as an interactive shell, it just did the right thing.

What you should have done first is "cat > fidsift.sh", then ascii-dumped it. Then pressed Ctrl-D after it was done to tell cat that you're done sending. The problem with this is that the empeg's serial port uses no flow control, so if there's a buffer overrun or similar, it has no way to compensate, and you'll just lose data. xmodem or similar would be a lot better, but I don't think that there's any such stuff on the empeg to use.
Posted by: Roger

Re: help! fidsift - 20/09/2004 10:23

Quote:
xmodem or similar would be a lot better, but I don't think that there's any such stuff on the empeg to use.


The developer image supports ZModem transfers. How do you think we used to get the player binaries on there ourselves?
Posted by: mlord

Re: Oh no! A serious flaw! - 20/09/2004 12:05

Quote:
Could this line be added to config.ini to run fidsift from HiJack?


Ahh... yes. And thank you for reminding me. I had been meaning to add that to my own config.ini once @;MENUEXEC was implemented.. like now!

Cheers
Posted by: SE_Sport_Driver

Re: Oh no! A serious flaw! - 20/09/2004 22:07

Mark, should I only have folders in my fids folder after running this? I see a bunch of loose fids in there after running it from the HiJack menu... Does the player software need to be turned off or does a "rw" need to be sent prior?
Posted by: mlord

Re: Oh no! A serious flaw! - 21/09/2004 04:51

Ahhh.. yes, it probably has trouble running if the player s/w is active. Bummer.. but one could write a fancier shell wrapper for it, that does a seek and destroy on the player before running fidsift, and then restarts the player again after.

Bitt? (I'm overloaded here)
Posted by: wfaulk

Re: Oh no! A serious flaw! - 21/09/2004 14:25

Gosh, I don't have an empeg shell available right now, but it'd be something like:

Code:
kill `ps ax | grep player | grep -v grep | awk '{print $1;}'`

fidsift.sh
/player/bin/player


I'll double check that soon. I'm sure it's wrong right now (I don't remember the correct player path, or know that awk is installed, or, for that matter, that the empeg's ps puts PID in the first column), but should give someone a start. It should also be possible to fit it into one line for MENUEXEC, but I haven't done any research into how much I can throw at it. I think '&&'s work, so you should just be able to combine those three lines with them.

Anyway, I'll get back to you.
Posted by: mlord

Re: Oh no! A serious flaw! - 21/09/2004 15:54

I'm not sure that there's a ps command normally available. If not, then it'll have to parse through /proc/nnn/* to find the player threads.

Cheers
Posted by: mcomb

Re: Oh no! A serious flaw! - 21/09/2004 16:49

Quote:
I'm not sure that there's a ps command normally available.


There is a ps, but no awk, split, eval, etc to do the parsing. This seems to work...

Code:

kill `for file in /proc/*/cmdline; do grep -v self $file>/dev/null && grep player $file>/dev/null && echo ${file:6:2} ; done`



but I am sure one of you can come up with something cleaner. Anyway, you still have the problem that the player automatically relaunches from init anytime you kill it.

-Mike
Posted by: wfaulk

Re: Oh no! A serious flaw! - 21/09/2004 17:07

ps is there, at least on the developer install. I forgot, though, that it lists each thread as a different process. And awk is not available. Nor is tail. sed is, though. I'll have to figure out a good script for it. Also, you have to kill -2, otherwise it just restarts the player app. I'll get back to it later today.
Posted by: Daria

Re: Oh no! A serious flaw! - 21/09/2004 19:15

Quote:
ps is there, at least on the developer install. I forgot, though, that it lists each thread as a different process. And awk is not available. Nor is tail. sed is, though. I'll have to figure out a good script for it. Also, you have to kill -2, otherwise it just restarts the player app. I'll get back to it later today.


In the old pthreads (clone()) world, each thread gets a pid. The new world has some problems, it seems.
Posted by: image

Re: Oh no! A serious flaw! - 22/09/2004 02:00

someone can just compile busybox really quick and then you can have the killall command. i did it some time ago, but i can't seem to find the binary, and i don't have my player w/ me. maybe tommorow.
Posted by: SE_Sport_Driver

Re: Oh no! A serious flaw! - 22/09/2004 09:44

Mark, I forgot to mention (and was unable to measure) but the last hard drive upgrade that I did went WAY faster than I expected it to. And it was the largest yet - copying 60gb of data to an 80gb drive. Do you think sifted fids could have helped here? I started mid-evening and didn't have to wait for the copy over night per usual.

So even if people start using Jemplode's rebuld on PC option, it appears that fidsift has other benefits in player performance.
Posted by: mlord

Re: Oh no! A serious flaw! - 22/09/2004 12:39

That all makes very good sense -- the kernel in the Empeg just does linear searches through the directory when opening a file, and if the directory is very large, it will have to do multiple seeks/reads and longer searches to find each file..

But newer drives are faster nowadays too, which could also be a factor.

Cheers
Posted by: wfaulk

Re: Oh no! A serious flaw! - 22/09/2004 14:51

Eh, you just need to kill the lowest-numbered PID and all's well. No need to try to kill all of them.
Posted by: image

Re: Oh no! A serious flaw! - 23/09/2004 12:52

true. but the reason i said to use killall was because you don't have to specify the pid#, therefore achieving your goal trying to use awk, sed, etc. of course, if you have busybox, you'd have those utils, so...

btw, here is busybox
Posted by: SE_Sport_Driver

Re: Oh no! A serious flaw! - 15/10/2004 20:34

Anyone have any luck with this?
Posted by: ukengb

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 09/02/2005 08:15

Hope Mark is still following this thread.

What are the rules for the naming of these subdirs?

I can see that fidsift.sh chops the fid into directory and file components, but is this the required modus operandi?

Reason I ask is that I'm looking into creating my own local 'fids' structure that I can then simply rsync up to the empeg (sadly mp3tofid is not quite suitable for my requirements) and so I can create whatever directory structure I want.

Can the subdirs be any name, or do they have to be in the format that Mark creates with fidsift? I guess it's dependent on how the player searches for files and about that I have no idea.

How many levels of subdirs can there be? I assume only this one, but have to ask.

Ta.
Posted by: Roger

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 09/02/2005 08:38

The structure must be the same as fidsift.sh builds it.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 09/02/2005 12:22

Yeah, what roger said!

The subdir names are very simple: _xxxxx
where "xxxxx" are the most significant 5 digits (hex) of the 8-digit original "fid" (file identifier). The files within each subdir are then named using the remaining 3 least significant digits. IF the original fid file had fewer than 8 digits, it must first be padded with zeros on the left to get an 8-digit value to then do the 5:3 split on.

Cheers
Posted by: ukengb

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 09/02/2005 15:24

Thanks for the info. From the specific way fidsift chopped them up I guessed it was important.

Thanks again.
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 05/03/2005 20:55

Is there a way to have Hijack sense a sync on the player and run fidsift before the player ap restarts?
Posted by: tman

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 05/03/2005 21:39

Hijack already moves the files for you...
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 05/03/2005 21:45

Quote:
Hijack already moves the files for you...


*Usually*.. but I have noticed that it doesn't always intercept things with JEmplode..

Cheers
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 05/03/2005 22:11

Ah.. cool. I remember that being a feature, but I thought it was dropped at some point.

Also, it will only do it if the folders are already there... so if you added enough tracks to warrant a new folder, then it just dumps the fids into the root correct?
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 18/07/2005 18:18

Can we make this a Sticky?

I think this is an underrated tool.
Posted by: tonyc

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 18/07/2005 19:14

Done.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 18/07/2005 19:43

Can we please cool it with the stickies? They're to the point of being abused now, and I find it quite annoying to have to scroll down to see the real "new" threads everywhere.

Fidsift is (or should be) in the FAQ already, along with every other sticky thread here.

cheers
Posted by: SonicSnoop

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 18/07/2005 20:10

Maybe the stickied threads could be moved over to the FAQ folder or something that way they can still be stickied and easy to get too but dont clog the other folders up?
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 18/07/2005 20:11

Valid point, but you're supposed to be so dang flattered that I'd request a sticky for your work that you'd blush.

Maybe this can fall under the "3rd Party Software" thread that is stickied.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 18/07/2005 21:56

Thanks Brad,

But yeah, how about perhaps a BBS category for nothing but "sticky" posts that link back to the original threads?

I'm confident there must be some way to accomplish this.

??
Posted by: tonyc

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 18/07/2005 22:14

Unstickified, along with the SYLT and emphatic threads.

No good deed goes un-bitched about.
Posted by: cushman

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 19/07/2005 00:25

Quote:
But yeah, how about perhaps a BBS category for nothing but "sticky" posts that link back to the original threads?

We have a FAQ section of this BBS that has 4 entire posts devoted to it, maybe we could use that as an interim BBS->The One True FAQ stepping point for some of these stickied threads. Process would be:

1. Thread is deemed important enough to sticky
2. Copy or link of thread is included in FAQ section of the BBS (can you have a mirror of the thread in two board sections? This would be ideal)
3. When someone gets a round tuit, the FAQ BBS entry is entered into the riocar.org FAQ or subsection (like the 3rd party software thread) and removed from the FAQ section of the BBS - original thread is maintained, linked to from the riocar.org FAQ.
Posted by: zexpe

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 30/10/2005 21:09

I've completely missed out on this interesting development. Would love to give it a go, and have just been wading my way through the old threads to make sure I've understood everything about fidsift.

First off, why no mention of fidsift in the FAQ yet?

Is Hijack + jEmplode + fidsift combination still not a 100% reliable and fully-functioning machine yet?

Is the latest version of the script sitting on a CVS or a webpage somewhere... or should I trust the fact that I've got found the most update version attached to some random post in this forum? ;-)

Finally... the following question doesn't seem to have been answered yet...

Quote:
Ah.. cool. I remember that being a feature, but I thought it was dropped at some point.

Also, it will only do it if the folders are already there... so if you added enough tracks to warrant a new folder, then it just dumps the fids into the root correct?


...and the answer's quite important as it makes the difference between fidsift being a run-once and forget task and a run every once in a while task.

Also, I guess all the above only applies with v2 player software. I am correct in thinking the v3 software (in whatever state it is in now, alpha or beta) does this all automatically anyway?

Cheers,
Ross
Posted by: SE_Sport_Driver

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 16/10/2006 12:13

I thought I'd give this thread a bump because it's being mentioned in the maxfid thread. One small side effect I found this weekend was that sorted fids are NOT compatible with version 1 of the player software. I was doing a drive upgrade on a friend's player and bootted his player up from the slave drive to test something out. Suddenly, all of his music and playlists were gone on the slave drive when using the player! Simply putting 2.00final on the player fixed that however. Just a little FYI.

PS - Anyone have any idea why searching for "fidsift" or "fidsift.sh" in the Subject field doesn't find this thread? I've tried to find it the last few days and was having trouble.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 16/10/2006 12:31

Quote:

PS - Anyone have any idea why searching for "fidsift" or "fidsift.sh" in the Subject field doesn't find this thread? I've tried to find it the last few days and was having trouble.


Heh. I get just about a 1% success rate when using the BBS search for anything here.

But Google appears to be indexing us again, and entering "fidsift.sh" at Google.ca quickly finds this thread now.

Cheers
Posted by: tman

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 16/10/2006 14:11

Quote:
I thought I'd give this thread a bump because it's being mentioned in the maxfid thread. One small side effect I found this weekend was that sorted fids are NOT compatible with version 1 of the player software.

The feature only got added into the player codebase in v2 so thats not too surprising...

I don't think it is even fully implemented in v2. It supports reading from the various directories but you need Mark's script and/or kernel to do the writing part.
Posted by: Roger

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 16/10/2006 18:06

Quote:
I don't think it is even fully implemented in v2.


From my page, under "FID Subdirectories":

Quote:
When the player was still at version 1.0, the FID files were all in the same directory (except when they were on separate disks, of course). As the number of files in a directory increases, the performance worsens.

At some point in the v2.0 beta releases, the player began supporting a slightly different layout for these two directories. In order to improve performance, the files can now be put into subdirectories.

...

This new layout is supported by the v2.0 players when looking for files, but when writing them, it uses the old, v1.0-compatible layout.

In v3.0, it writes the files to the new layout, but supports the old layout, to ensure that it works on players still using the old layout.
Posted by: wfaulk

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 16/10/2006 22:29

Quote:
Anyone have any idea why searching for "fidsift" or "fidsift.sh" in the Subject field doesn't find this thread?

Works perfectly fine for me.
Posted by: drakino

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 20/10/2006 00:53

Worked fine for me too. Clicked search, put in "fidsift" and checked the subject instead of subject and body, Filled the first page of results.
Posted by: mlord

Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs - 28/05/2007 00:36

Here (attached) is my latest copy of fidsift.sh

This one has a cosmetic "bug" fix, and is also more tolerant of being interrupted and then re-run afterwards, not that I recommend doing so.

Cheers