#251728 - 15/03/2005 14:46
CGI Scripts in Hijack, and other fancies
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
Here's M.Cushman's original PM to me: Quote:
Hi Mark, I've been messing around on and off with a project to give a "party" web interface that would let the user see the current playing order of tracks and append/insert new tracks intothe playlist.
The hurdle that I've been getting around is getting the current playlist. I modified some of tonyc's code to look into the dynamic data partition and pull out fids 'around' the current track. I can then represent this list of fids in XML and let the browser build the rest of the interface.
In order to deliver this XML file to the browser, I must have a way to send it over http - so I can either roll my own webserver with the code to get/form the XML playlist or deliver it viakhttpd if it supported CGI scripts. I have not written many web servers ;-) but how hard wouldit be to allow CGI scrips in khttpd? We could identify CGI scripts as files marked executable with a .cgi extension, or have config.ini lines with allowed extensions. Only allow GET, no POST. On a GET request, pass the URL past the ? to the executable as the single argument, execute the binary, and pass the output back to the browser.
This could be useful for more than just a web interface, too. My new Palm has Bluetooth and a BT interface to the Empeg could use the current running playlist information just as easily as a web browser.Anyway, let me know what you think.
|
|
Top
|
|
|
|
|
#251729 - 15/03/2005 14:50
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
And here is my reply to the question of adding CGI capability to Hijack's web server.
It's not too difficult to add basic CGI program support (without seatbelts). Mostly a matter of having the server (khttpd) recognize the .cgi URL, and set up arg lists for an execve() one-way-branch to execute the program. Before doing the execve(), it must also reassign (dup) the open socket's file descriptor to stdout for the .CGI program.
But I think we're rapidly arriving at the time where a new distro is needed, with more userspace tools (like a more complete web server) and stuff. Mind you, that userspace webserver will initially lack the kernel's "playlist browser" capability (and .xml support etc..), but I think we can find a way to provide that eventually.
I'm currently near-150% tied up until April, when I hope to roll a new Empeg distro for my own use, and eventually release it here.
My current plan is to begin with the buildroot kit from Eric Anderson (Busybox + uclibc) and go from there. The empeg player binary and it's own libraries will have to live inside there, perhaps chroot'd.
The Hijack FTP server will definitely live on, as it is still the simplest way to get new and/or replacement software into a player. I may even extend it to support standard PASV (passive) FTP while I'm at it.
Anyway, that's what I hope to do. Others may beat me to it, of course.
Cheers
|
|
Top
|
|
|
|
|
#251730 - 15/03/2005 15:27
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
veteran
Registered: 21/01/2002
Posts: 1380
Loc: Erie, CO
|
Mark, this sounds good. It makes sense to move the webserver out into userspace, and I'm sure we can find a way to provide the XML interfaces there. I'll work on completing the rest of my project - I don't think it will be done util at least April anyway (lots of snowboarding left to do) and we'll see what you come up with then. This is exciting, lots of new/better software for the Empeg.
|
|
Top
|
|
|
|
|
#251731 - 15/03/2005 15:28
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
carpal tunnel
Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
|
I'm very much in favor of having a standard Busybox/uclibc-based distribution available on the empeg. However, which "more complete" web server would we be talking about? More complete suggests more resource-hungry. Aside from CGI support, what else do we need from an empeg web server that we don't get already from khttpd? I would prefer not to have extra resources taken up to get the same webserver functionality we have right now, and I'm not familiar with any "lightweight" httpds out there that would give us anything like what we get from khttpd, without significant modifications.
If you do decide to go this direction, can you keep khttpd around, optionally compiled in for those of us who would like to stick with it?
|
|
Top
|
|
|
|
|
#251732 - 15/03/2005 15:31
Re: CGI Scripts in Hijack, and other fancies
[Re: tonyc]
|
veteran
Registered: 21/01/2002
Posts: 1380
Loc: Erie, CO
|
Pulling it out of the kernel would still keep it lightweight - and in pageable memory too. We could even have multiple web server binaries that you could choose from depending on features that you want. I just want khttpd with CGI, nothing more - others might want a more featured httpd but it would be their choice to run or not.
|
|
Top
|
|
|
|
|
#251733 - 15/03/2005 15:50
Re: CGI Scripts in Hijack, and other fancies
[Re: cushman]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31569
Loc: Seattle, WA
|
But I like the simplicity of "install the empeg .upgrade file, then install hijack" and you're done.
I really depend on having that simple webserver that's already in there. It's just so nice as it is, even though it's limited.
Is there some way we can do both?
|
|
Top
|
|
|
|
|
#251734 - 15/03/2005 15:50
Re: CGI Scripts in Hijack, and other fancies
[Re: tonyc]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
I'm thinking of just using the httpd (and telnetd etc..) that are already built-into BusyBox. Basic CGI is supported, and lots of other nice things in that toolkit.
Cheers
|
|
Top
|
|
|
|
|
#251735 - 15/03/2005 15:55
Re: CGI Scripts in Hijack, and other fancies
[Re: tfabris]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
Quote:
But I like the simplicity of "install the empeg .upgrade file, then install hijack" and you're done.
I really depend on having that simple webserver that's already in there. It's just so nice as it is, even though it's limited.
Is there some way we can do both?
Oh.. important point here: I hope to have the new distro install in the same footprint as the existing empeg stuff, without the need to repartition anything.
This new distro would ideally get packaged (by me) as a .upgrade file, with everything necessary included. Just apply the .upgrade (over serial or ethernet), and you're done: now running the new distro, new kernel, new webserver, etc..
And one can still apply further .upgrade files to revert back or forward after that.
This distro would obsolete the current player .upgrade files -- we'll need a simple tool to take those and repackage them inside the new distro, and an independent tool to simply install a new player package onto an existing installation over HTTP/FTP.
The possibilities are almost limitless, and it is all rather simple to accomplish.
|
|
Top
|
|
|
|
|
#251736 - 15/03/2005 15:57
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
carpal tunnel
Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
|
Another drawback that tman reminded me of is that this httpd would now be running in userspace, where the greedy player would be starving it by running at realtime priority. The good thing about having khttpd in the kernel is we get good response times. When you're running in userland, the player wins, and everyone else loses. I would imagine a significant negative effect on the responsiveness of the web interface if the httpd moves to user space.
|
|
Top
|
|
|
|
|
#251737 - 15/03/2005 15:58
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31569
Loc: Seattle, WA
|
Quote:
This distro would obsolete the current player .upgrade files -- we'll need a simple tool to take those and repackage them inside the new distro
Or, perhaps, our friends in Cambridge could simply give you permission to redistribute the player app binary, so that the end users don't have to repackage things on their own end?
|
|
Top
|
|
|
|
|
#251738 - 15/03/2005 16:04
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
Note that this also means dumping the current closed-source empeg "init" program, which then makes it much simpler to stop/start the player, remount drives ro/rw, etc..
This can lead to such nice things as a FAST database rebuild running on the empeg to replace/supplement the way it (doesn't) work now.
And on-the-fly switching among multiple player versions, without even a reboot required.
And possibly installing Das U-Boot in flash (in place of the kernel), and having it load the kernel from disk. No more flash reprogramming needed for anything..
And an ext3 filesystem as the default, maybe. Gotta look at the RAM usage again, but I'd really want it on my 32MB players. Have it normally running r/o as now, but with the capability to switch to r/w for database updates and other s/w needs, and then back to r/o again, perhaps all automatically. EDIT: I suppose that could also be done for ext2 on lessor-memory players
Cheers
Edited by mlord (15/03/2005 16:07)
|
|
Top
|
|
|
|
|
#251739 - 15/03/2005 16:16
Re: CGI Scripts in Hijack, and other fancies
[Re: tonyc]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
Quote:
Another drawback that tman reminded me of is that this httpd would now be running in userspace, where the greedy player would be starving it by running at realtime priority. The good thing about having khttpd in the kernel is we get good response times. When you're running in userland, the player wins, and everyone else loses. I would imagine a significant negative effect on the responsiveness of the web interface if the httpd moves to user space.
The only scheduler difference between an all-kernelspace thread and one which runs in userspace is that, in userspace, it can be preempted mid-stride, whereas in kernelspace (on these kernels) it is never preempted.
Apart from that, the real-time priority player app still gets scheduled before the webserver, even if it's a kernel based webserver.
I don't think we'll see much of a difference due to the scheduler.
But if we leave it as pageable, then there may be some times when it has to be paged back into memory, and it will be slower whenever that happens. I doubt this will be common, though, and a simple mlockall() "fixes" it for those willing to expend the RAM on it.
Cheers
|
|
Top
|
|
|
|
#251740 - 16/03/2005 06:43
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
addict
Registered: 11/01/2002
Posts: 612
Loc: Reading, UK
|
Hi I know I don't contribute much - I just lurk via the email summaries  Anyway, this sounds interesting; As well as 2 Empegs, I also have the Sharp Zaurus and am looking at their toolchain and build solution at the minute: OpenEmbeddedGiven that OE also stands for OpenEmpeg, I was wondering if this would be a suitable candidate? One of the main reasons for suggesting it is to try and lower the bar for developers to be able to access the Empeg. Although oe is a fairly complex package, it's basically a download and cut'n'paste (and wait!) job to get a complete distro built (including x-compile toolchain, kernel, Qt, libc, ulibc, apache, perl etc) - or at least it should be! Now I now you don't want most of these packages on the empeg but the ease of use principle is sound. There's also an oe wiki which could easily host OpenEmpeg build info until such time... lbt
_________________________
LittleBlueThing
Running twin 30's
|
|
Top
|
|
|
|
|
#251741 - 16/03/2005 11:33
Re: CGI Scripts in Hijack, and other fancies
[Re: LittleBlueThing]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
That's a lot of what buildroot also does.. basically type 'make' and it creates the cross-compiler toolchain from scratch, downloading full sources etc.., and then builds the target root filesystem from scratch, again downloading full sources etc..
But perhaps OE has stuff beyond that which could be useful. dunno, but I'll check!
For most empeg s/w, the need is for a properly setup GCC toolchain. Once a person has that (downloadable from Hijack site, btw), then regular Linux packages should be really really easy to build, simply using 'make'.
Cheers
|
|
Top
|
|
|
|
|
#251742 - 28/03/2005 01:03
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
veteran
Registered: 21/01/2002
Posts: 1380
Loc: Erie, CO
|
Mark, you mentioned in another thread that your distro would be targeting players with upgraded memory, is this the only option (to have more RAM installed)? Any possibility of getting a more stripped-down distro for 16M players?
|
|
Top
|
|
|
|
|
#251743 - 28/03/2005 01:16
Re: CGI Scripts in Hijack, and other fancies
[Re: cushman]
|
addict
Registered: 29/06/2002
Posts: 531
Loc: Triangle, VA
|
What would be nice, not sure how possible it is, but have some kind of gui when going to install ask how much ram you have and then install stuff based on that. but im sure thats probobly alot more work then its worth.
_________________________
-D
Modifying and Tweaking is a journey,
not a destination................................
MKIIa : 60gig - 040103286 - Blue - v2 + PCATS tuner
MKIIa : 20gig - 040103260 - Blue - v3a8 + Mark Lord Special Edition Cherry Dock
|
|
Top
|
|
|
|
|
#251744 - 28/03/2005 01:32
Re: CGI Scripts in Hijack, and other fancies
[Re: cushman]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
The biggest two differences will likely be:
(1) a newer C-library, so when the player is running both versions will be (at least partially) loaded into memory. Cost: 0.5MB or so
(2) Player runs from a *real* init program, so init will be bigger and in memory at the same time as the player. cost : another 0.2MB or so, or less.
(3) Any servers that are running will cost more memory than before.
That's the basic memory footprint issue. Leave out the above three, and one has the current player software -- no point in a new distro for that.
Cheers
|
|
Top
|
|
|
|
|
#251745 - 28/03/2005 01:45
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
veteran
Registered: 21/01/2002
Posts: 1380
Loc: Erie, CO
|
Ok, so here is a dump of /proc/meminfo while playing a song on my player: Code:
total: used: free: shared: buffers: cached:
Mem: 15269888 14770176 499712 1286144 331776 2383872
Swap: 0 0 0
MemTotal: 14912 kB
MemFree: 488 kB
MemShared: 1256 kB
Buffers: 324 kB
Cached: 2328 kB
SwapTotal: 0 kB
SwapFree: 0 kB
This is with hijack, telnetd and empire running. The MemFree fluctuates but doesn't seem to get much above 0.5M. Is there a good way to tell how much memory I have available for other apps/libraries and when it will start affecting the player software? I guess if there is enough justification to doing so I'll upgrade my player (because I want a new web server) but I don't think I've ever seen a thread that talks about the memory footprint of the player/hijack/etc.
|
|
Top
|
|
|
|
|
#251746 - 28/03/2005 10:54
Re: CGI Scripts in Hijack, and other fancies
[Re: cushman]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
*Any* memory use by the kernel or other apps affects the player. Period.
But to find out how much additional memory Hijack consumes, one would need to measure free memory just prior to the kernel invoking "init", and compare with a non-Hijack kernel. But of course, not even that is completely accurate, since the web/ftp threads require/use extra memory (about 12KB per thread, I think) when they're in-use, though not otherwise.
Perhaps the best place to display usage would be from a simple ;@EXEC_ONCE app, to be run after all other ;@EXEC lines from config.ini.
To compare that with a non-Hijacked player would be difficult, though. Maybe I can build it into both a Hijack and non-hijack kernel -- a simple printk() to dump out available memory from the same point (just prior to returning to the player from reading config.ini in kernel space).
Cheers
|
|
Top
|
|
|
|
|
#251747 - 28/03/2005 10:55
Re: CGI Scripts in Hijack, and other fancies
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14481
Loc: Canada
|
OOoo..
Extra planned memory footprint for the new distro: ext3 crash-proof filesystem as standard feature. Probable cost of a megabyte or so.
Cheers
|
|
Top
|
|
|
|
|
|
Previous Topic
Index