Hijack v344: fids subdirectories support

Hijack v344 is now available.

New in this version:

(1) FTP Filesystem source code is in the kernel source tree, but NOT configured or compiled in by default. No bloat that way, but it's still there for people like me who build their own kernels. This will eventually move to the RioReceiver kernels.

(2) Added a new Hijack config.ini option: trace_fs=1 When set, this option causes all file accesses to be logged via the serial port. Very useful for seeing what the player thinks it is doing at any point in time..

(3) More comprehensive support for /empeg/fids?/_xxxxx/ subdirectories. Hijack now detects the "new" fids (tunes & playlists) layout in /empeg/fids?/*, and automatically redirects ALL accesses to the appropriate subdirectory, if one exists. This means that if you convert your fids subdirs to this new format using the attached script, fidsift.sh (download and run it on your player, once ever), Hijack will ensure that all subsequent sync operations respect and use the new structure, whether they want to or not!

Background: the player software (v2 and v3) already supports the new structure, but has a deficiency whereby it normally does not download new tunes into the subdirs, even when such subdirs already exist. For playback and database rebuilds, the player software DOES look in the subdirs, as well as the top level /empeg/fids?/ directories. This and subsequent hijack versions now force the player software to always use the subdirectories if they exist, and ONLY the subdirectories -- preventing accidental duplication and confusion when the same file name may exist in both places.

Switching your player over to the new structure is EASY -- just download and run the attached script once on your player. Running it more than once is okay too, but not needed (harmless).

EDIT: script is available here.

Why do this? It can dramatically speed up access to tunes on a heavily loaded player (thousands of tracks, or even tens of thousands of tracks), cutting "sync" times by 20-50% or more. It also speeds up the Hijack web interface, and helps the player slightly during normal music playback.

Why not do it? If you ever upload new tracks to the player later without Hijack installed, then things will work just fine -- until you reinstall Hijack. At that point, nothing bad happens, but you will need to re-run the attached script before the uploaded tunes can be accessed through Hijack. Hardly a blip for anyone, I suspect.

I would especially like to see Paul try this out.

Cheers

Background: the player software (v2 and v3) already supports the new structure, but has a deficiency whereby it normally does not download new tunes into the subdirs, even when such subdirs already exist.

I think that only applies to v2 -- doesn't v3 always put new tunes into the subdirectories, unless it's replacing another file with the same FID?

Peter

Sure looks like it. Running v3, fidsifted, made a (delete/create) change, still looks fine (nothing other than dirs in /drive?/fids/)

What I observed was that the player was happy to re-use "available" FIDs in existing subdirs, but when it ran out of those it would NOT create a new higher numbered subdir for subsequent downloads. Hijack now does that for it.

EDIT: not sure about v3 -- most of my testing yesterday was with v2. Takes a while to test this too -- JEmplode is too clever about spotting duplicates for me!

Cheers

JEmplode is too clever about spotting duplicates for me

uncheck dedupe in options. hash won't be created

Greetings!

Busy day. I just stumbled on the thread. I will take a look at it later this evening or this weekend. Things are a bit crazy at the moment, but I will give it a shot.

Hello

Can someone explain me exactly how to run this script? Step by step? Please...

Thanks in advance....

bye

See this thread:
http://empeg.comms.net/php/showflat.php?Cat=&Board=hackers_prog&Number=182162&page=0&view=collapsed&sb=5&o=7&fpart=

Thanks!

just download and run the attached script once on your player. Running it more than once is okay too, but not needed (harmless).

Ok, as a linux newbie, what should I do precisely? FTP the script somewhere in my empeg, then enter the command prompt via serial by poressing the Q at boot, and then?

Thank you

This is what I did on an Xp machine
Connect player to serial
Start empeg
Upload new HiJack 344
Open FTP client (FTP)
Open Hyperterminal (HT)
Set RW in FTP - site RW
send file to player root /
Connect to player via HT then press q [enter]
Send chmod 0755 fidsift.sh command via HT
Set RO in FTP - site RO
Type QUIT in HT
Press ctrl+c in HT
Type /fidsift.sh into HT
Wait until folders are written in HT.
End HT
End FTP
Reboot empeg


If this is wrong or i missed something let me know.

Sprry, I did not notoce other messages had already answered to my question.
I still have a problem: I can't enter the command prompt. Q does not seem to work. Same with CTRL-C. Same with the Quit menu on the empeg.
WHen shoudl I press Q/CTRL-V ?

I can't enter the command prompt.

Then please look here.

you will need the Developers version of the player software to access the player, I think..

Edited to remove apostrophe

Sean in NC

I actually did, Tony, and I have entered the command prompt several times in the past, altough never with v. 3.0b3a, which I am running now.
I see the boot messages as I recycle the power to the player, but somehow it does not seem to get input fro the keyboard. If select Quit from the empeg menu, it actually tells me this:

Restored terminal settings
Remounting first music partition read-only
Remounting second music partition read-only
Player exited normally: 0
Switching to shell-player loop
Starting bash.
empeg:/empeg/bin#

but at that point nothing that I type shows on screen, and the empeg simply stays there and wait. :? What am I doing wrong?


Edit:
And yes, I do have the developer version.

And flow control is off?

Mark, is the version of your script in this thread newer than the second on in your original thread? (swapon/swapoff)?

Yes, the version linked in this thread is the newest: v1.03, with swapon/swapoff included.

Cheers

Worked great for me. Haven't done a sync yet but the Hijack web interface is indeed noticably snappier. Thanks, Mark!

Well, my player is not exactly Grzelakian:

2 60GB drives (111GB usable)
14709 tracks
8.72GB free

Pre-sift synchronization time: 8:47
fidsift execution time: 24:45
Post-sift synchronization time: 4:35

48% reduction! Awesome. It seems like the player loads the music database significantly faster on boot, too (I wish I had done before & after timings of that process). Haven't experimented with navigation, etc.

Thank you again, Mark!

Bedtime...

Jim
PS: Everyone should do this to their player. In fact, I think we should move or copy this entire thread to General, since I doubt everyone reads Programming.

Edit: While the capacity may not be Grzelakian, the drives are! My player has 2 of Paul's old 60GB drives...

And flow control is off?

That was it! How could I forget? I did this tens of times... Sorry guys, too much work I guess... Thank you!

Edit: script run, it took more or less 10 minuts on my 3500-songs collection. All is working well and I'll try now to notice improvements...

Greetings!

I ran into delays last night. I will do this upgrade this weekend!

Jimmy, I too wish that I had timed the "Loading Music Database" on boot up, maybe Paul can do this for us.

Mark, is it necessary to ftp this file on the player? If we have to be connected via serial to execute the script, can't we just direct connect to the player in serial, do a rw, then send the file (like we'd send a anna-vu logo), do a "chmod 755 filename.fs", type RO and quit? Seems like it'd save time.

Some people might be scared by this because they don't want to ftp, but this might be easier.

Any way you can get the file on the player is good.
Nowadays, people are probably more used to "put"ting files with ftp than using rz/sz...

/Michael

Yeah, I suppose FTP might scare some people off, but the UNIX command line will scare them off even more. I figure those folks probably don't have the developer image or hijack anyway.

I used the player to do a bunch of playlist adds and navigating while on the way to work today. The whole player seems somewhat more responsive. The sync times under 5 minutes are the biggest advantage, though.

Jim

The database files themselves are mostly unaffected by this change -- sure, I mighta shaved a few milliseconds off (dubious), but it cannot be measured by the human eye. However, in addition to reading the database files at that time, the player also does a very quick scan of the fids directories. That part could be as much as a second or so quicker now (for loaded players), but I've not tried to measure it.

EDIT: and how did I shave a few milliseconds away? Well, because Hijack is now FORCING redirection to the subdirs, attempts to access files in the FIDS directories now succeed on the first syscall for each, rather than requiing two syscalls from the player (one for the old location, then one for the new location..).

Cheers

Well, OK. I was probably just delighted about the sync time and my "world view" was effected. :-)

Like I said, I wish I had timed it. Anyhow, great job and thanks!

Greetings!

And so it begins... I will edit this post as I go along. How are you timing your syncs, and under what conditions?

-------------------
Empeg State:
Version 2.0 Final, Developer, Hijack 344
2 x 80GB Hard Drives (160GB nominal, 149GB useable)
27066 Tracks (as defined by soup view in emplode)
6.4 GB Free (as defined by status bar in emplode)
73EF <29679 Decimal> Highest FID # (via serial port examination, leaving off last digit)

Pre:
Power-on time = 21 seconds : (timed from power applied by pushing into dock until standby mode entered)
Power-on time = 16.2 seconds : (using "time player" from shell, and quitting player as soon as possible)
Load player into emplode = 7 minutes 46 seconds : (using serial connect - slower, but known constant transfer rate)

Groan - one of my music filesystems (where I also hold utilities and stuff) needs an FSCK. I need to resolve that before getting any sync times... While this occurs (45 min per drive), does anyone have timing tests they would like?

Greetings!

So, after cleaning out the drives with an FSCK, I can continue... Also, I have a clone player (working on LabRat, clone is Archive) that I can use to get comparisons and timings after the fact. They are (well, were) in perfect rsync.

-------------------
Empeg State:
Version 2.0 Final, Developer, Hijack 344
2 x 80GB Hard Drives (160GB nominal, 149GB useable)
27066 Tracks (as defined by soup view in emplode)
6.4 GB Free (as defined by status bar in emplode)
73EF <29679 Decimal> Highest FID # (via serial port examination, leaving off last digit)

Before:
Power-on time = 21 seconds : (timed from power applied by pushing into dock until standby mode entered)
Power-on time = 16.2 seconds : (using "time player" from shell, and quitting player as soon as possible)
Load player into emplode = 7 minutes 46 seconds : (using serial connect - slower, but known constant transfer rate)
Sync = infinity * : (emplode using serial connect - deleted one song from one playlist)
Database Rebuild = 21 minutes 8 seconds : (going into shell, setting drives read/write and running the player application to rebuild database files)
FSCK Time = 61 minutes 2 seconds : (time fsck -fay /drive0, assuming clean drive to begin with)

During:
fidsift.sh = less than 1 hour (time ./fidsift.sh) - Sorry. The first time around, the time command did not return a value to me. My PC went into a full system virus scan, and my hyperterm results became choppy. So, trusting that I would see useful results, I wandered away... The command succeeded, though.

After:
Power-on time = 23 seconds : (timed from power applied by pushing into dock until standby mode entered)
Power-on time = 19.7 seconds : (using "time player" from shell, and quitting player as soon as possible)
Load player into emplode = 7 minutes 48 seconds : (using serial connect - slower, but known constant transfer rate)
Sync = infinity * : (emplode using serial connect - deleted one song from one playlist)
Database Rebuild = 6 minutes 52 seconds : (going into shell, setting drives read/write and running the player application to rebuild database files)
FSCK Time = 60 minutes 12 seconds : (time fsck -fay /drive0, assuming clean drive to begin with)

* infinity - my players FID count seems to be giving emplode and database creation grief. I am able to upload the tracks with emplode, but further syncs seems to cause an error 0x8004003e from emplode. The "building databases" part of the sync goes by unnaturally fast, and then Emplode crashes during the "writing player configuration" phase, causing a Dr. Watson dump. The player just sits there with "Building music databases..." on the screen. After over 30 minutes passed, the building database phase appeared to complete, and the player application completed its load. Going in with a serial link, the /drive0/var directory has no database files. I will rebuild them by hand as above... After that rebuild, the player works normally, with all changes made from the sync in place. It is just an inconvenience. No other problems noticed yet from beyond beyond max-fid-count. See this thread for more information.

WOW!!! I did not expect to see that kind of improvement, given that I am so far out there with the FID count. Note: Since normal syncs do not work for me, I am probably a bad example for this... All times that involve human interaction (quitting player, watching the clock) are dependent on human response time, and are all +/- a second or two...

*bump*

To change the "last updated" time for the forum, and mark the thread as having a new item.

Your bump should mention that the message it's a reply to was updated

I guessed that was why and went back and read it. Impressive.

Valid point. I forgot that bit.

I was impressed by the database build improvements. Much faster!!!

I'm bummed out that you can't do a sync from emplode still...

Well, with the audiobooks, I know that if I actually worked on it, I could condense the ones with 99 files per CD to something smaller, and basically clean and rebuild the player so that it did not have that many FIDs on it. But it does not clear the problem.

Basically, it seems that emplode launches the database rebuild, but that some error is passed or something happens where emplode thinks it finished insanely quickly. It then moves on to the next step (writing player configuration) and fails with a timeout. While all this is happening, the player is rebuilding the database files, but never actually succeeds in writing them to disk, even though it goes through without a sign of an error.

In all honesty, I have not played the most recently added files much. When I tested with a few random known "last files", though, it seemed to work okay.

Most impressive player stats, Paul!

Good to see the database rebuild times are somewhat improved by this change! As in they now take around 33% of the time they previously required!

Pity about Emplode. I suppose JEmplode probably survives this test though..

Cheers

Um, actually I have not tried jemplode recently. I usually sync over USB to the player from Windows 2000. When I tried that, long ago, the combination did not work well at all. So I have been using emplode all along.

I can add tracks and all, and (to be honest) I do not sync much. I have my playlists fairly static at this point, and the rare random change is not that big a deal.

Edit: correcting typos.

(2) 30GB HDD
2.0 developer
emplode via ethernet
Hijack 344
10235 Tracks

---Before fidsift.sh---

21 seconds: boot up time to standby from power off
8 seconds: load database into emplode
4m57s rebuild database - sync (deleted a track from a playlist)

---After fidsift.sh---

21 seconds: boot up time to standby from power off
5 seconds: load database into emplode
2m20s rebuild database - sync (deleted a track from a playlist)

A 47% improvement in sync time!!!

Awesome work Mark!!!

Also, using Emplode for some strange reason when I sync for the very first time after having read the db, the sync goes through but does not complete (as in, no database rebuild, changes made in emplode are not reflected after database is read again). Subsequent syncs work fine. I have become used to just syncing the first time when I get in to emplode, but it would be nice if I didn't have to do this as it wastes time. Any ideas?

I used to have various issues syncing, so now every time I go to make any changes (new music, corrections etc) I first telnet in and kill off everything I find using ps that isn't needed. Some of these are obvious like emphatic, but there are occasionally others, depending on what I have on the machine.

After doing that syncs work fine. Admittedly I only have about 7000 fids so not a huge collection.

Greetings!

This is more likely based on the open processes. If any of them have log files (or anything else for that matter) locked open on the standard root or music partitions, emplode (jemplode) are unable to dismount them as part of their normal sync process.

Shouldn't those programs be set to turn off when detecting a sync?

Yes - I should probably have plonked these apps elsewhere, but never mind. Maybe when I rebuild with 3.0 beta (I'm not risking it until at least beta stage) I'll rebuild a more sensible structure.