Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 1 of 3 1 2 3 >
Topic Options
#183410 - 07/10/2003 17:12 fidsift.sh -- rearrange /drive?/fids/ into subdirs
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Quite some time ago, the player software was enhanced to be able to deal with /fids/ directories that are broken up into smaller subdirs. This can speed up database rebuilds and filesystem checks etc, and is a worthwhile thing to have.

The Emplode and emptool software still just upload everything into /drive?/fids/, but the player software (and Hijack) can handle things in either format, including a mix of the two methods.

So.. here's my small script for (re)sifting the tune and tag files into appropriate subdirs, on a one or two drive RioCar/Empeg player.

To use, upload via FTP to /fidsort.sh, and set the exec permissions:

ftp myplayer
site rw
cd /
put fidsift.sh
chmod 0755 fidsift.sh
site ro
quit
Then connect via the serial port, hit control^C to gain control of the player, and do this command: /fidsift.sh

Then go and have coffee or cola, and when you get back it will eventually finish, and you can view the results in /drive0/fids/ and /drive1/fids/. As new tunes are later uploaded, unsifted files will acculumate again, and you might want to periodically rerun the script in the same way as the first time, to sift the newly uploaded tunes into their subdirs (not necessary, but nice).

The script is attached -- EDIT: there's a better version later in this thread..

Cheers



Attachments
182162-fidsift.sh (860 downloads)



Edited by mlord (07/10/2003 20:29)

Top
#183411 - 07/10/2003 19:24 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Want me to do a "before and after" timing of database rebuild? ~22,000 tracks I think.
_________________________
Brad B.

Top
#183412 - 07/10/2003 19:41 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Now that could be very educational!

I'm also wondering if the script will just crap out with that many fids.. go ahead and give it a whirl!

(it won't actually damage anything if it does crap out.. maybe I'll try a test here..)

Top
#183413 - 07/10/2003 19:49 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
I trust that if it craps, only part of the fids will be moved and emplode will just handle it?
_________________________
Brad B.

Top
#183414 - 07/10/2003 19:51 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Actually, the place where it might crap would just result in nothing happening.

And if it did only do it partially, no big deal -- emplode doesn't know nor care, and neither does the player or hijack. Everything still just works

Top
#183415 - 07/10/2003 19:52 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Hang on a few minutes.. I'm generating a dummy fids directory with 22000 "tracks"..

Top
#183416 - 07/10/2003 19:56 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
I just tried it (on my laptop) with 25000 fids (over 50000 files). No problem. Not tried on the player (would take me too long), but it's safe.

Cheers

Top
#183417 - 07/10/2003 20:03 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
can hijack be modified to run this script automatically after uploading? like i would love it before the player restarts, so the player would still be expected to be in rw mode.

Top
#183418 - 07/10/2003 20:16 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: image]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Seems like it might make (already long) sync's longer.
_________________________
Brad B.

Top
#183419 - 07/10/2003 20:16 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Okay, here's a slightly improved version.

This one shrinks the /fids/ directory afterwards to improve lookups further, and also copes correctly with a /fids/ directory that contains no files.

Cheers


Attachments
182190-fidsift.sh (904 downloads)


Top
#183420 - 07/10/2003 20:17 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Faster updates than HiJack!
_________________________
Brad B.

Top
#183421 - 07/10/2003 20:18 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
This script, after the initial run, is actually quite fast. So it wouldn't really slow down most syncs at all (not noticeably), unless uploading several hundred tracks.

I thought about building it into hijack, but a better option is just to bind it into the menu system as a user app.. Gotta write that new menu thingie Real Soon Now (the "site exec" stuff and friends).

Cheers

Top
#183422 - 07/10/2003 20:26 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
chmod 0755 = what does this equate to?

I'm using WS_FTP and it doesn't do numbered chmods... only checkboxes.
_________________________
Brad B.

Top
#183423 - 07/10/2003 20:27 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
rwxr-xr-x == 0755

User(owner): read/write/execute
Group: read/execute
Other: read/execute


Edited by mlord (07/10/2003 20:27)

Top
#183424 - 07/10/2003 20:39 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Thanks for the edit too.
_________________________
Brad B.

Top
#183425 - 07/10/2003 20:52 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Update: The FidSift is currently taking longer to run (at least this first time) than the "pre sift" sync... will it be a photo finish or is there a long way to go?
_________________________
Brad B.

Top
#183426 - 07/10/2003 21:05 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
The first sift will be slow. But you should be getting the odd progress indication in the form of a mkdir: created directory `_00000' style of message once in a while.

Cheers

Top
#183427 - 07/10/2003 21:33 Way to go Mark! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Okay, I have 20,919 mp3s on the player (I always forget how many I have because the player doesn't always removed duplicates when repeating the root playlist like it used to) with 6.26GB free of 130GB. Anyway... the process:

1. Installed latest Hijack.

2. Connected via emplode and added one 4.54mb file.

3. Pre "FidSift" Sync
Start: 10:11
End: 10:28
Elapsed: 0:17 (I was surprised it was this quick!)

4. Installed and ran FidSift.
Started sifting drive0: 10:52
Started sifting drive1: 11:11
Ended: 11:34
Elapsed: 0:42
Drive0 went up to about _0004d and Drive1 went to _0005b. As Mark pointed out, this shouldn't take this long after the initial run.

5. Connected via emplode, replaced same 4.54MB file.

6. Post "FidSift" Sync
Start: 11:35
End: 11:43
Elaspsed: 0:08

That's over a 100% speed increase!


Edited by SE_Sport_Driver (07/10/2003 21:43)
_________________________
Brad B.

Top
#183428 - 07/10/2003 21:36 Re: Way to go Mark! [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Ahh, good. I was hoping for some kind of measurable improvement there. I forgot to do the "before" timings on my own players, and I have not written a script to convert the files back to the old naming/layout scheme, so..

WooHoo!

(thanks for trying it out)

Top
#183429 - 07/10/2003 21:38 Re: Way to go Mark! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
No, thank you! Database rebuilds were painfully slow but I can live with 8 minutes now! I assume that everyone can get rebuilds to be over twice as fast too.
_________________________
Brad B.

Top
#183430 - 07/10/2003 21:40 Re: Way to go Mark! [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Mmm.. I suppose I could have the script do both drives in parallel, which would cut the time for the first sift down by quite a bit. But that's a one-time only event (per player), so no big deal. And I don't have a two-drive unit here to test with, so I think I'll just leave well enough alone!

A subsequent "sift" should be near instantaneous, unless a thousand or so (or more) tracks are modified with emplode/emptool.

Cheers

Top
#183431 - 07/10/2003 21:45 Re: Way to go Mark! [Re: SE_Sport_Driver]
loren
carpal tunnel

Registered: 23/08/2000
Posts: 3826
Loc: SLC, UT, USA
Sweeet. Nice job Mark. So can we expect this to be built into hijack soon? If so i think i'll wait. I'm going to be completely rebuilding my player soon anyhow. It would rock if this could be run after a certain number of tracks have been added. Could Jemplode do this somehow??
_________________________
|| loren ||

Top
#183432 - 07/10/2003 21:49 Re: Way to go Mark! [Re: loren]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Mmm.. Well, JEmplode over ethernet could use FTP for the file transfers, in which case it could just use the new subdir structure as it uploads tracks. Maybe Mike will add that someday.

I don't know about building it into Hijack -- so don't wait for that to happen. Once I someday implement the "run a command from the menu" feature it will be trivial to do. But I keep getting distracted here..

This is pretty simple to do as-is.. and even if it's only done once it will still make a difference. The only tricky bit is getting the script onto the player, but that only need be done the first time. From then on, it's still there and can be run just by hitting control^c and typing it's name (using the serial link). And it's only slow the first time!

Sleep time..

Cheers

Top
#183433 - 08/10/2003 00:47 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
Foz
member

Registered: 24/10/2000
Posts: 106
Loc: San Jose, CA
Is this thing safe under the V3 alpha kernel?

-- Gary F.
_________________________
Eeyore, Original Owner -- Mk II 80 Gb, Blue S/N #090000803 Tigger, 2nd Owner -- Mk IIa, 80 Gb, Blue S/N #40103789

Top
#183434 - 08/10/2003 01:18 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: Foz]
simspos
enthusiast

Registered: 28/03/2002
Posts: 230
Loc: Dudley, UK
Nice work Mark,.... people of the world (with obscenely large music collections) Rejoice!

Top
#183435 - 08/10/2003 05:06 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: Foz]
johnmcd3
enthusiast

Registered: 19/04/2001
Posts: 369
Loc: Seattle, WA (formerly Houston,...
Yes.
_________________________
1998 BMW ///M3 30 GB Mk2a, Tuner, and 10 GB backup

Top
#183436 - 08/10/2003 06:17 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: Foz]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
The "v3alpha kernel" is the same as the v2-final kernel, so.. yes, it's safe.

It is also just fine with the v3alpha player software, too.

-ml

Top
#183437 - 08/10/2003 08:34 Oh no! A serious flaw! [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
Okay, there's a flaw with this scheme.

When syncing with Emplode or emptool, the player stores new/modified FIDS in the top level /driveX/fids/ directory, rather than the subdirs. Same when doing deletes. Nothing wrong with that, other than some duplication.

BUT.. the player software does NOT search the top level directory if it finds a subdir.. OUCH. I was under the impression that it was smart enough to look in the right order, but it is actually even smarter than that -- smart enough not to "waste" the effort. This is a good thing, but it means that you MUST re-run fidsift.sh after each sync(Ugh!), and even that won't properly delete things.

Okay, I'll fix this, in the Hijack kernel, so that the player does "the right thing" whether it wants to or not. Later today, or tomorrow.

The plan is have the kernel automatically remap ALL /fids/ accesses to the subdirs when they (subdirs) exist. This way, syncs which affect existing fids will work correctly. New uploads can still be dropped into the top level /fids/ directories, if no appropriate subdir exists at the time. Not an issue for now.

Cheers

Top
#183438 - 08/10/2003 09:56 Re: Oh no! A serious flaw! [Re: mlord]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
my solution.

;@AC ;@EXEC /bin/fidsift.sh

cant use ;@EXEC_ONCE because that only executes after a reboot.

unless you uploaded over 500 files, the time diff is unnoticable.

Top
#183439 - 08/10/2003 10:33 Re: Oh no! A serious flaw! [Re: image]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14472
Loc: Canada
That will almost work. Requires a re-sync after the first sync, to ensure the database is rebuilt using the newly sifted files.. thus the problem.

Cheers

Top
Page 1 of 3 1 2 3 >