Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 1 of 3 1 2 3 >
Topic Options
#183410 - 07/10/2003 17:12 fidsift.sh -- rearrange /drive?/fids/ into subdirs
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Quite some time ago, the player software was enhanced to be able to deal with /fids/ directories that are broken up into smaller subdirs. This can speed up database rebuilds and filesystem checks etc, and is a worthwhile thing to have.

The Emplode and emptool software still just upload everything into /drive?/fids/, but the player software (and Hijack) can handle things in either format, including a mix of the two methods.

So.. here's my small script for (re)sifting the tune and tag files into appropriate subdirs, on a one or two drive RioCar/Empeg player.

To use, upload via FTP to /fidsort.sh, and set the exec permissions:

ftp myplayer
site rw
cd /
put fidsift.sh
chmod 0755 fidsift.sh
site ro
quit
Then connect via the serial port, hit control^C to gain control of the player, and do this command: /fidsift.sh

Then go and have coffee or cola, and when you get back it will eventually finish, and you can view the results in /drive0/fids/ and /drive1/fids/. As new tunes are later uploaded, unsifted files will acculumate again, and you might want to periodically rerun the script in the same way as the first time, to sift the newly uploaded tunes into their subdirs (not necessary, but nice).

The script is attached -- EDIT: there's a better version later in this thread..

Cheers



Attachments
182162-fidsift.sh (860 downloads)



Edited by mlord (07/10/2003 20:29)

Top
#183411 - 07/10/2003 19:24 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Want me to do a "before and after" timing of database rebuild? ~22,000 tracks I think.
_________________________
Brad B.

Top
#183412 - 07/10/2003 19:41 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Now that could be very educational!

I'm also wondering if the script will just crap out with that many fids.. go ahead and give it a whirl!

(it won't actually damage anything if it does crap out.. maybe I'll try a test here..)

Top
#183413 - 07/10/2003 19:49 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
I trust that if it craps, only part of the fids will be moved and emplode will just handle it?
_________________________
Brad B.

Top
#183414 - 07/10/2003 19:51 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Actually, the place where it might crap would just result in nothing happening.

And if it did only do it partially, no big deal -- emplode doesn't know nor care, and neither does the player or hijack. Everything still just works

Top
#183415 - 07/10/2003 19:52 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Hang on a few minutes.. I'm generating a dummy fids directory with 22000 "tracks"..

Top
#183416 - 07/10/2003 19:56 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
I just tried it (on my laptop) with 25000 fids (over 50000 files). No problem. Not tried on the player (would take me too long), but it's safe.

Cheers

Top
#183417 - 07/10/2003 20:03 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
can hijack be modified to run this script automatically after uploading? like i would love it before the player restarts, so the player would still be expected to be in rw mode.

Top
#183418 - 07/10/2003 20:16 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: image]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Seems like it might make (already long) sync's longer.
_________________________
Brad B.

Top
#183419 - 07/10/2003 20:16 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Okay, here's a slightly improved version.

This one shrinks the /fids/ directory afterwards to improve lookups further, and also copes correctly with a /fids/ directory that contains no files.

Cheers


Attachments
182190-fidsift.sh (904 downloads)


Top
#183420 - 07/10/2003 20:17 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Faster updates than HiJack!
_________________________
Brad B.

Top
#183421 - 07/10/2003 20:18 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
This script, after the initial run, is actually quite fast. So it wouldn't really slow down most syncs at all (not noticeably), unless uploading several hundred tracks.

I thought about building it into hijack, but a better option is just to bind it into the menu system as a user app.. Gotta write that new menu thingie Real Soon Now (the "site exec" stuff and friends).

Cheers

Top
#183422 - 07/10/2003 20:26 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
chmod 0755 = what does this equate to?

I'm using WS_FTP and it doesn't do numbered chmods... only checkboxes.
_________________________
Brad B.

Top
#183423 - 07/10/2003 20:27 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
rwxr-xr-x == 0755

User(owner): read/write/execute
Group: read/execute
Other: read/execute


Edited by mlord (07/10/2003 20:27)

Top
#183424 - 07/10/2003 20:39 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Thanks for the edit too.
_________________________
Brad B.

Top
#183425 - 07/10/2003 20:52 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Update: The FidSift is currently taking longer to run (at least this first time) than the "pre sift" sync... will it be a photo finish or is there a long way to go?
_________________________
Brad B.

Top
#183426 - 07/10/2003 21:05 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
The first sift will be slow. But you should be getting the odd progress indication in the form of a mkdir: created directory `_00000' style of message once in a while.

Cheers

Top
#183427 - 07/10/2003 21:33 Way to go Mark! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Okay, I have 20,919 mp3s on the player (I always forget how many I have because the player doesn't always removed duplicates when repeating the root playlist like it used to) with 6.26GB free of 130GB. Anyway... the process:

1. Installed latest Hijack.

2. Connected via emplode and added one 4.54mb file.

3. Pre "FidSift" Sync
Start: 10:11
End: 10:28
Elapsed: 0:17 (I was surprised it was this quick!)

4. Installed and ran FidSift.
Started sifting drive0: 10:52
Started sifting drive1: 11:11
Ended: 11:34
Elapsed: 0:42
Drive0 went up to about _0004d and Drive1 went to _0005b. As Mark pointed out, this shouldn't take this long after the initial run.

5. Connected via emplode, replaced same 4.54MB file.

6. Post "FidSift" Sync
Start: 11:35
End: 11:43
Elaspsed: 0:08

That's over a 100% speed increase!


Edited by SE_Sport_Driver (07/10/2003 21:43)
_________________________
Brad B.

Top
#183428 - 07/10/2003 21:36 Re: Way to go Mark! [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Ahh, good. I was hoping for some kind of measurable improvement there. I forgot to do the "before" timings on my own players, and I have not written a script to convert the files back to the old naming/layout scheme, so..

WooHoo!

(thanks for trying it out)

Top
#183429 - 07/10/2003 21:38 Re: Way to go Mark! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
No, thank you! Database rebuilds were painfully slow but I can live with 8 minutes now! I assume that everyone can get rebuilds to be over twice as fast too.
_________________________
Brad B.

Top
#183430 - 07/10/2003 21:40 Re: Way to go Mark! [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Mmm.. I suppose I could have the script do both drives in parallel, which would cut the time for the first sift down by quite a bit. But that's a one-time only event (per player), so no big deal. And I don't have a two-drive unit here to test with, so I think I'll just leave well enough alone!

A subsequent "sift" should be near instantaneous, unless a thousand or so (or more) tracks are modified with emplode/emptool.

Cheers

Top
#183431 - 07/10/2003 21:45 Re: Way to go Mark! [Re: SE_Sport_Driver]
loren
carpal tunnel

Registered: 23/08/2000
Posts: 3826
Loc: SLC, UT, USA
Sweeet. Nice job Mark. So can we expect this to be built into hijack soon? If so i think i'll wait. I'm going to be completely rebuilding my player soon anyhow. It would rock if this could be run after a certain number of tracks have been added. Could Jemplode do this somehow??
_________________________
|| loren ||

Top
#183432 - 07/10/2003 21:49 Re: Way to go Mark! [Re: loren]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Mmm.. Well, JEmplode over ethernet could use FTP for the file transfers, in which case it could just use the new subdir structure as it uploads tracks. Maybe Mike will add that someday.

I don't know about building it into Hijack -- so don't wait for that to happen. Once I someday implement the "run a command from the menu" feature it will be trivial to do. But I keep getting distracted here..

This is pretty simple to do as-is.. and even if it's only done once it will still make a difference. The only tricky bit is getting the script onto the player, but that only need be done the first time. From then on, it's still there and can be run just by hitting control^c and typing it's name (using the serial link). And it's only slow the first time!

Sleep time..

Cheers

Top
#183433 - 08/10/2003 00:47 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: mlord]
Foz
member

Registered: 24/10/2000
Posts: 106
Loc: San Jose, CA
Is this thing safe under the V3 alpha kernel?

-- Gary F.
_________________________
Eeyore, Original Owner -- Mk II 80 Gb, Blue S/N #090000803 Tigger, 2nd Owner -- Mk IIa, 80 Gb, Blue S/N #40103789

Top
#183434 - 08/10/2003 01:18 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: Foz]
simspos
enthusiast

Registered: 28/03/2002
Posts: 230
Loc: Dudley, UK
Nice work Mark,.... people of the world (with obscenely large music collections) Rejoice!

Top
#183435 - 08/10/2003 05:06 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: Foz]
johnmcd3
enthusiast

Registered: 19/04/2001
Posts: 369
Loc: Seattle, WA (formerly Houston,...
Yes.
_________________________
1998 BMW ///M3 30 GB Mk2a, Tuner, and 10 GB backup

Top
#183436 - 08/10/2003 06:17 Re: fidsift.sh -- rearrange /drive?/fids/ into sub [Re: Foz]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
The "v3alpha kernel" is the same as the v2-final kernel, so.. yes, it's safe.

It is also just fine with the v3alpha player software, too.

-ml

Top
#183437 - 08/10/2003 08:34 Oh no! A serious flaw! [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Okay, there's a flaw with this scheme.

When syncing with Emplode or emptool, the player stores new/modified FIDS in the top level /driveX/fids/ directory, rather than the subdirs. Same when doing deletes. Nothing wrong with that, other than some duplication.

BUT.. the player software does NOT search the top level directory if it finds a subdir.. OUCH. I was under the impression that it was smart enough to look in the right order, but it is actually even smarter than that -- smart enough not to "waste" the effort. This is a good thing, but it means that you MUST re-run fidsift.sh after each sync(Ugh!), and even that won't properly delete things.

Okay, I'll fix this, in the Hijack kernel, so that the player does "the right thing" whether it wants to or not. Later today, or tomorrow.

The plan is have the kernel automatically remap ALL /fids/ accesses to the subdirs when they (subdirs) exist. This way, syncs which affect existing fids will work correctly. New uploads can still be dropped into the top level /fids/ directories, if no appropriate subdir exists at the time. Not an issue for now.

Cheers

Top
#183438 - 08/10/2003 09:56 Re: Oh no! A serious flaw! [Re: mlord]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
my solution.

;@AC ;@EXEC /bin/fidsift.sh

cant use ;@EXEC_ONCE because that only executes after a reboot.

unless you uploaded over 500 files, the time diff is unnoticable.

Top
#183439 - 08/10/2003 10:33 Re: Oh no! A serious flaw! [Re: image]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
That will almost work. Requires a re-sync after the first sync, to ensure the database is rebuilt using the newly sifted files.. thus the problem.

Cheers

Top
#183440 - 08/10/2003 12:20 Re: Oh no! A serious flaw! [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Mmm.. now that I've traced through it some, it appears that the player IS doing the RIGHT THING(tm), at least some of the time. I need to play with it more to make sure.

At database rebuild time, it correctly looks for (eg.) /drive0/fids/100 before it looks for the same file in a subdirectory (eg.) /drive0/fids/_00000/100

So, as long as it remembers where it found each file for later (part of the database?), then it should be fine as is.

I'll check that now. Meanwhile, I have a hacked hijack version that automatically ensures that new uploads end up in the right places. Not released yet.

Cheers

Top
#183441 - 08/10/2003 12:44 Re: Oh no! A serious flaw! [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Yeah, it's fine.

My worries were for naught. All is well.

But I may still release the changes I made to Hijack, which ensure that the /fids/ directory stays sifted after the initial run.. But then again, a userspace tool can still do this equally well, so why bloat Hijack?

Cheers

Top
#183442 - 08/10/2003 15:54 Re: Oh no! A serious flaw! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
I agree that there is no point in bloating HiJack. Could this user ap be executed via ethernet or ftp rather than serial?
_________________________
Brad B.

Top
#183443 - 08/10/2003 16:35 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
loren
carpal tunnel

Registered: 23/08/2000
Posts: 3826
Loc: SLC, UT, USA
Bloat hijack because it's one less thing we'll have to install and keep track of. And it'll be easier to run. Right? Wrong?
_________________________
|| loren ||

Top
#183444 - 08/10/2003 19:50 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
Daria
carpal tunnel

Registered: 24/01/2002
Posts: 3937
Loc: Providence, RI
Think of this as technical prose, and not really a good solution.

telnet empeg
rw
mv /empeg/bin/player /empeg/bin/player.disabled
cat > /empeg/bin/player << __EOF__
#!/bin/sh
echo "init will helpfully reboot if it can't run the player. Bad init. No biscuit."
return 0
__EOF__
chmod +x /empeg/bin/player
ps auxww|grep player|grep -v grep|awk '{print $2}'|xargs kill
sleep 3
mv /empeg/bin/player.disabled /empeg/bin/player
fidsift
kill -1 1

Top
#183445 - 09/10/2003 03:25 Re: Oh no! A serious flaw! [Re: mlord]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4172
Loc: Cambridge, England
At database rebuild time, it correctly looks for (eg.) /drive0/fids/100 before it looks for the same file in a subdirectory (eg.) /drive0/fids/_00000/100

So, as long as it remembers where it found each file for later (part of the database?), then it should be fine as is.
It doesn't remember; it re-does the search every time.

Peter

Top
#183446 - 09/10/2003 05:15 Re: Oh no! A serious flaw! [Re: peter]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5680
Loc: London, UK
Yeah, but at database rebuild time, it does a directory-recursive search for FID files. At FID-lookup time, it has a specific strategy -- which might be the other way round, I don't recall.
_________________________
-- roger

Top
#183447 - 09/10/2003 05:44 Re: Oh no! A serious flaw! [Re: mlord]
Shonky
pooh-bah

Registered: 12/01/2002
Posts: 2009
Loc: Brisbane, Australia
OK Mark I just tried it on a 9436 song empeg - about 50Gb.

My force database rebuild consisted of changing the name of one playlist.

pre sift sync : 4:53
initial sift time was 18:27
post sift sync : 3:20

So not 100% like Brad but still faster. It obviously helps players with more files.

However, there was one problem! On my /drive1 it created a dodgy directory. This is a section of the inital sift...
Sifting /drive1/fids..

mkdir: created directory `_00000'
mkdir: created directory `_0'
mkdir: created directory `_00001'
mkdir: created directory `_00010'
mkdir: created directory `_1'
mkdir: created directory `_00011'
mkdir: created directory `_00012'
mkdir: created directory `_00013'
mkdir: created directory `_00014'
mkdir: created directory `_00015'
hijack: found new-style fids subdirectories
mkdir: created directory `_00016'
mkdir: created directory `_00017'
mkdir: created directory `_00018'


In the _0 directory is now a file called 10 which is the info type file. In the _1 directory is now a file 11 which is the matching mp3 file for the info file in the _0 directory.

According to a CSV export of my database (done after this song was upload I'm 100% certain) the fid of this track should have been 7172 decimal i.e. 0x1C04. So I have no idea what has happened and where it got possible 110 and 111 as the fids from. I don't understand regexps that well, so I can't see anything obviously wrong with your script.

I tried moving the files back to where they were and called them 1c040 and 1c041. I reran the sift program but this time it worked fine and put the files in the correct directory. So I have no idea what went wrong. I haven't listened to the particular song in awhile so it may not have been stored properly, although I didn't receive any sort of database errors via emplode.

Anyway thanks for doing it. It still sped up my rebuilds.
_________________________
Christian
#40104192 120Gb (no longer in my E36 M3, won't fit the E46 M3)

Top
#183448 - 09/10/2003 05:57 Re: Oh no! A serious flaw! [Re: Roger]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Yeah, I think it is the other way around. Not a big deal, that, though it can be the source of some confusion. Database rebuilds look for non-subdir'd files first, then for the subdirectory versions. At playback time, the player looks into the subdirectories first, and then for the flat file versions. I think.

Rather than theoretically "fixing" that someday, I would very much prefer that a future release of the player software merely include code to automatically write new FIDS to the subdirectories when uploading. Unconditionally would be fine, but one could gate it by requiring that at least one subdir already exists or that the player be empty.

I have modified Hijack-v344 to automatically redirect ALL acceseses to the subdirs, player or otherwise, but only when performed through the symlinks in /empeg/fids?/ (not when done via /drive?/fids/, but only if at least one (any) subdir already exists. I didn't bother with the "empty player" case. To be released soon.

Cheers

Top
#183449 - 09/10/2003 06:08 Re: Oh no! A serious flaw! [Re: Shonky]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Mmm.. Shonky, that's rather odd -- and theoretically impossible from the way the script is written. I suppose maybe /bin/bash got low on memory or something. I will modify the script to turn on swap before/after running, to alleviate any such memory pressures.

The upcoming Hijack-v344 kernel includes code to automatically keep a sifted drive sifted after future sync operations. This code also replaces some existing code that was in the http server, so overall it doesn't contribute much bloat at all.

Cheers

Top
#183450 - 09/10/2003 12:33 Re: Oh no! A serious flaw! [Re: Daria]
Daria
carpal tunnel

Registered: 24/01/2002
Posts: 3937
Loc: Providence, RI
everything except the kill -1 1 at the end (which i hoped would make init respawn the player) worked. fidsifted without use of a serial port last night.

Top
#183451 - 23/11/2003 12:54 Re: Oh no! A serious flaw! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Bump. I thought I'd bump this thread for people interested in fidsift now that the new HiJack is coming.
_________________________
Brad B.

Top
#183452 - 19/09/2004 22:24 help! fidsift [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
I decided to send fidsift.sh via TeraTerm (Hyperterminal type program) rather than ftp'ing since it would save me the extra step of hooking up an ethernet cable.

I did an "ro" and "rom" before changing to the root directory and using the "Send File" option.

Problem is, once I sent the file, it appears to have executed the file! Is this normal? It "looks" to be doing the right thing, but I didn't have a chance to be sure... I also never ran "chmod 0755 fidsift.sh".

Do I have anything to worry about? Just let it run? Why did it execute?

UPDATE: Looks like everything ran okay... but I see no trace of fidsift.sh being on the player... It opened the file but never saved it.. I've since ftp'd it to the root directory so it can be run in the future.


Edited by SE_Sport_Driver (19/09/2004 22:38)
_________________________
Brad B.

Top
#183453 - 19/09/2004 22:30 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Could this line be added to config.ini to run fidsift from HiJack?

[hijack]
;@MENUEXEC fidsift /fidsift.sh
_________________________
Brad B.

Top
#183454 - 19/09/2004 23:15 Re: help! fidsift [Re: SE_Sport_Driver]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
You probably sent an ascii dump rather than transferring the file via xmodem or similar, and since a .sh file is just a series of commands to the shell being run as an interactive shell, it just did the right thing.

What you should have done first is "cat > fidsift.sh", then ascii-dumped it. Then pressed Ctrl-D after it was done to tell cat that you're done sending. The problem with this is that the empeg's serial port uses no flow control, so if there's a buffer overrun or similar, it has no way to compensate, and you'll just lose data. xmodem or similar would be a lot better, but I don't think that there's any such stuff on the empeg to use.
_________________________
Bitt Faulk

Top
#183455 - 20/09/2004 10:23 Re: help! fidsift [Re: wfaulk]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5680
Loc: London, UK
Quote:
xmodem or similar would be a lot better, but I don't think that there's any such stuff on the empeg to use.


The developer image supports ZModem transfers. How do you think we used to get the player binaries on there ourselves?
_________________________
-- roger

Top
#183456 - 20/09/2004 12:05 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Quote:
Could this line be added to config.ini to run fidsift from HiJack?


Ahh... yes. And thank you for reminding me. I had been meaning to add that to my own config.ini once @;MENUEXEC was implemented.. like now!

Cheers

Top
#183457 - 20/09/2004 22:07 Re: Oh no! A serious flaw! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Mark, should I only have folders in my fids folder after running this? I see a bunch of loose fids in there after running it from the HiJack menu... Does the player software need to be turned off or does a "rw" need to be sent prior?
_________________________
Brad B.

Top
#183458 - 21/09/2004 04:51 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Ahhh.. yes, it probably has trouble running if the player s/w is active. Bummer.. but one could write a fancier shell wrapper for it, that does a seek and destroy on the player before running fidsift, and then restarts the player again after.

Bitt? (I'm overloaded here)

Top
#183459 - 21/09/2004 14:25 Re: Oh no! A serious flaw! [Re: mlord]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Gosh, I don't have an empeg shell available right now, but it'd be something like:

Code:
kill `ps ax | grep player | grep -v grep | awk '{print $1;}'`

fidsift.sh
/player/bin/player


I'll double check that soon. I'm sure it's wrong right now (I don't remember the correct player path, or know that awk is installed, or, for that matter, that the empeg's ps puts PID in the first column), but should give someone a start. It should also be possible to fit it into one line for MENUEXEC, but I haven't done any research into how much I can throw at it. I think '&&'s work, so you should just be able to combine those three lines with them.

Anyway, I'll get back to you.
_________________________
Bitt Faulk

Top
#183460 - 21/09/2004 15:54 Re: Oh no! A serious flaw! [Re: wfaulk]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
I'm not sure that there's a ps command normally available. If not, then it'll have to parse through /proc/nnn/* to find the player threads.

Cheers

Top
#183461 - 21/09/2004 16:49 Re: Oh no! A serious flaw! [Re: mlord]
mcomb
pooh-bah

Registered: 31/08/1999
Posts: 1649
Loc: San Carlos, CA
Quote:
I'm not sure that there's a ps command normally available.


There is a ps, but no awk, split, eval, etc to do the parsing. This seems to work...

Code:

kill `for file in /proc/*/cmdline; do grep -v self $file>/dev/null && grep player $file>/dev/null && echo ${file:6:2} ; done`



but I am sure one of you can come up with something cleaner. Anyway, you still have the problem that the player automatically relaunches from init anytime you kill it.

-Mike
_________________________
EmpMenuX - ext3 filesystem - Empeg iTunes integration

Top
#183462 - 21/09/2004 17:07 Re: Oh no! A serious flaw! [Re: mlord]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
ps is there, at least on the developer install. I forgot, though, that it lists each thread as a different process. And awk is not available. Nor is tail. sed is, though. I'll have to figure out a good script for it. Also, you have to kill -2, otherwise it just restarts the player app. I'll get back to it later today.
_________________________
Bitt Faulk

Top
#183463 - 21/09/2004 19:15 Re: Oh no! A serious flaw! [Re: wfaulk]
Daria
carpal tunnel

Registered: 24/01/2002
Posts: 3937
Loc: Providence, RI
Quote:
ps is there, at least on the developer install. I forgot, though, that it lists each thread as a different process. And awk is not available. Nor is tail. sed is, though. I'll have to figure out a good script for it. Also, you have to kill -2, otherwise it just restarts the player app. I'll get back to it later today.


In the old pthreads (clone()) world, each thread gets a pid. The new world has some problems, it seems.

Top
#183464 - 22/09/2004 02:00 Re: Oh no! A serious flaw! [Re: Daria]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
someone can just compile busybox really quick and then you can have the killall command. i did it some time ago, but i can't seem to find the binary, and i don't have my player w/ me. maybe tommorow.

Top
#183465 - 22/09/2004 09:44 Re: Oh no! A serious flaw! [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Mark, I forgot to mention (and was unable to measure) but the last hard drive upgrade that I did went WAY faster than I expected it to. And it was the largest yet - copying 60gb of data to an 80gb drive. Do you think sifted fids could have helped here? I started mid-evening and didn't have to wait for the copy over night per usual.

So even if people start using Jemplode's rebuld on PC option, it appears that fidsift has other benefits in player performance.
_________________________
Brad B.

Top
#183466 - 22/09/2004 12:39 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
That all makes very good sense -- the kernel in the Empeg just does linear searches through the directory when opening a file, and if the directory is very large, it will have to do multiple seeks/reads and longer searches to find each file..

But newer drives are faster nowadays too, which could also be a factor.

Cheers

Top
#183467 - 22/09/2004 14:51 Re: Oh no! A serious flaw! [Re: image]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Eh, you just need to kill the lowest-numbered PID and all's well. No need to try to kill all of them.
_________________________
Bitt Faulk

Top
#183468 - 23/09/2004 12:52 Re: Oh no! A serious flaw! [Re: wfaulk]
image
old hand

Registered: 28/04/2002
Posts: 770
Loc: Los Angeles, CA
true. but the reason i said to use killall was because you don't have to specify the pid#, therefore achieving your goal trying to use awk, sed, etc. of course, if you have busybox, you'd have those utils, so...

btw, here is busybox


Attachments
233743-busybox.tar.bz2 (506 downloads)


Top
#183469 - 15/10/2004 20:34 Re: Oh no! A serious flaw! [Re: SE_Sport_Driver]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Anyone have any luck with this?
_________________________
Brad B.

Top
#183470 - 09/02/2005 08:15 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
Hope Mark is still following this thread.

What are the rules for the naming of these subdirs?

I can see that fidsift.sh chops the fid into directory and file components, but is this the required modus operandi?

Reason I ask is that I'm looking into creating my own local 'fids' structure that I can then simply rsync up to the empeg (sadly mp3tofid is not quite suitable for my requirements) and so I can create whatever directory structure I want.

Can the subdirs be any name, or do they have to be in the format that Mark creates with fidsift? I guess it's dependent on how the player searches for files and about that I have no idea.

How many levels of subdirs can there be? I assume only this one, but have to ask.

Ta.

Top
#183471 - 09/02/2005 08:38 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: ukengb]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5680
Loc: London, UK
The structure must be the same as fidsift.sh builds it.
_________________________
-- roger

Top
#183472 - 09/02/2005 12:22 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: ukengb]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Yeah, what roger said!

The subdir names are very simple: _xxxxx
where "xxxxx" are the most significant 5 digits (hex) of the 8-digit original "fid" (file identifier). The files within each subdir are then named using the remaining 3 least significant digits. IF the original fid file had fewer than 8 digits, it must first be padded with zeros on the left to get an 8-digit value to then do the 5:3 split on.

Cheers

Top
#183473 - 09/02/2005 15:24 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
ukengb
member

Registered: 30/04/2003
Posts: 136
Loc: United Kingdom
Thanks for the info. From the specific way fidsift chopped them up I guessed it was important.

Thanks again.

Top
#183474 - 05/03/2005 20:55 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Is there a way to have Hijack sense a sync on the player and run fidsift before the player ap restarts?
_________________________
Brad B.

Top
#183475 - 05/03/2005 21:39 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: SE_Sport_Driver]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Hijack already moves the files for you...

Top
#183476 - 05/03/2005 21:45 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: tman]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Quote:
Hijack already moves the files for you...


*Usually*.. but I have noticed that it doesn't always intercept things with JEmplode..

Cheers

Top
#183477 - 05/03/2005 22:11 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: tman]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Ah.. cool. I remember that being a feature, but I thought it was dropped at some point.

Also, it will only do it if the folders are already there... so if you added enough tracks to warrant a new folder, then it just dumps the fids into the root correct?
_________________________
Brad B.

Top
#183478 - 18/07/2005 18:18 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Can we make this a Sticky?

I think this is an underrated tool.
_________________________
Brad B.

Top
#183479 - 18/07/2005 19:14 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: SE_Sport_Driver]
tonyc
carpal tunnel

Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
Done.
_________________________
- Tony C
my empeg stuff

Top
#183480 - 18/07/2005 19:43 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: tonyc]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Can we please cool it with the stickies? They're to the point of being abused now, and I find it quite annoying to have to scroll down to see the real "new" threads everywhere.

Fidsift is (or should be) in the FAQ already, along with every other sticky thread here.

cheers

Top
#183481 - 18/07/2005 20:10 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SonicSnoop
addict

Registered: 29/06/2002
Posts: 531
Loc: Triangle, VA
Maybe the stickied threads could be moved over to the FAQ folder or something that way they can still be stickied and easy to get too but dont clog the other folders up?
_________________________
-D Modifying and Tweaking is a journey, not a destination................................ MKIIa : 60gig - 040103286 - Blue - v2 + PCATS tuner MKIIa : 20gig - 040103260 - Blue - v3a8 + Mark Lord Special Edition Cherry Dock

Top
#183482 - 18/07/2005 20:11 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
Valid point, but you're supposed to be so dang flattered that I'd request a sticky for your work that you'd blush.

Maybe this can fall under the "3rd Party Software" thread that is stickied.
_________________________
Brad B.

Top
#183483 - 18/07/2005 21:56 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Thanks Brad,

But yeah, how about perhaps a BBS category for nothing but "sticky" posts that link back to the original threads?

I'm confident there must be some way to accomplish this.

??

Top
#183484 - 18/07/2005 22:14 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
tonyc
carpal tunnel

Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
Unstickified, along with the SYLT and emphatic threads.

No good deed goes un-bitched about.
_________________________
- Tony C
my empeg stuff

Top
#183485 - 19/07/2005 00:25 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
cushman
veteran

Registered: 21/01/2002
Posts: 1380
Loc: Erie, CO
Quote:
But yeah, how about perhaps a BBS category for nothing but "sticky" posts that link back to the original threads?

We have a FAQ section of this BBS that has 4 entire posts devoted to it, maybe we could use that as an interim BBS->The One True FAQ stepping point for some of these stickied threads. Process would be:

1. Thread is deemed important enough to sticky
2. Copy or link of thread is included in FAQ section of the BBS (can you have a mirror of the thread in two board sections? This would be ideal)
3. When someone gets a round tuit, the FAQ BBS entry is entered into the riocar.org FAQ or subsection (like the 3rd party software thread) and removed from the FAQ section of the BBS - original thread is maintained, linked to from the riocar.org FAQ.
_________________________
Mark Cushman

Top
#183486 - 30/10/2005 21:09 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: SE_Sport_Driver]
zexpe
journeyman

Registered: 21/08/2002
Posts: 63
Loc: Edinburgh, UK
I've completely missed out on this interesting development. Would love to give it a go, and have just been wading my way through the old threads to make sure I've understood everything about fidsift.

First off, why no mention of fidsift in the FAQ yet?

Is Hijack + jEmplode + fidsift combination still not a 100% reliable and fully-functioning machine yet?

Is the latest version of the script sitting on a CVS or a webpage somewhere... or should I trust the fact that I've got found the most update version attached to some random post in this forum? ;-)

Finally... the following question doesn't seem to have been answered yet...

Quote:
Ah.. cool. I remember that being a feature, but I thought it was dropped at some point.

Also, it will only do it if the folders are already there... so if you added enough tracks to warrant a new folder, then it just dumps the fids into the root correct?


...and the answer's quite important as it makes the difference between fidsift being a run-once and forget task and a run every once in a while task.

Also, I guess all the above only applies with v2 player software. I am correct in thinking the v3 software (in whatever state it is in now, alpha or beta) does this all automatically anyway?

Cheers,
Ross

Top
#183487 - 16/10/2006 12:13 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
SE_Sport_Driver
carpal tunnel

Registered: 05/01/2001
Posts: 4903
Loc: Detroit, MI USA
I thought I'd give this thread a bump because it's being mentioned in the maxfid thread. One small side effect I found this weekend was that sorted fids are NOT compatible with version 1 of the player software. I was doing a drive upgrade on a friend's player and bootted his player up from the slave drive to test something out. Suddenly, all of his music and playlists were gone on the slave drive when using the player! Simply putting 2.00final on the player fixed that however. Just a little FYI.

PS - Anyone have any idea why searching for "fidsift" or "fidsift.sh" in the Subject field doesn't find this thread? I've tried to find it the last few days and was having trouble.
_________________________
Brad B.

Top
#183488 - 16/10/2006 12:31 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: SE_Sport_Driver]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Quote:

PS - Anyone have any idea why searching for "fidsift" or "fidsift.sh" in the Subject field doesn't find this thread? I've tried to find it the last few days and was having trouble.


Heh. I get just about a 1% success rate when using the BBS search for anything here.

But Google appears to be indexing us again, and entering "fidsift.sh" at Google.ca quickly finds this thread now.

Cheers

Top
#183489 - 16/10/2006 14:11 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: SE_Sport_Driver]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Quote:
I thought I'd give this thread a bump because it's being mentioned in the maxfid thread. One small side effect I found this weekend was that sorted fids are NOT compatible with version 1 of the player software.

The feature only got added into the player codebase in v2 so thats not too surprising...

I don't think it is even fully implemented in v2. It supports reading from the various directories but you need Mark's script and/or kernel to do the writing part.

Top
#183490 - 16/10/2006 18:06 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: tman]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5680
Loc: London, UK
Quote:
I don't think it is even fully implemented in v2.


From my page, under "FID Subdirectories":

Quote:
When the player was still at version 1.0, the FID files were all in the same directory (except when they were on separate disks, of course). As the number of files in a directory increases, the performance worsens.

At some point in the v2.0 beta releases, the player began supporting a slightly different layout for these two directories. In order to improve performance, the files can now be put into subdirectories.

...

This new layout is supported by the v2.0 players when looking for files, but when writing them, it uses the old, v1.0-compatible layout.

In v3.0, it writes the files to the new layout, but supports the old layout, to ensure that it works on players still using the old layout.
_________________________
-- roger

Top
#183491 - 16/10/2006 22:29 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: SE_Sport_Driver]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Quote:
Anyone have any idea why searching for "fidsift" or "fidsift.sh" in the Subject field doesn't find this thread?

Works perfectly fine for me.
_________________________
Bitt Faulk

Top
#183492 - 20/10/2006 00:53 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: wfaulk]
drakino
carpal tunnel

Registered: 08/06/1999
Posts: 7868
Worked fine for me too. Clicked search, put in "fidsift" and checked the subject instead of subject and body, Filled the first page of results.

Top
#183493 - 28/05/2007 00:36 Re: fidsift.sh -- rearrange /drive?/fids/ into subdirs [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14478
Loc: Canada
Here (attached) is my latest copy of fidsift.sh

This one has a cosmetic "bug" fix, and is also more tolerant of being interrupted and then re-run afterwards, not that I recommend doing so.

Cheers


Attachments
299723-fidsift.sh (830 downloads)


Top
Page 1 of 3 1 2 3 >