Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 2 of 8 < 1 2 3 4 5 6 7 8 >
Topic Options
#308500 - 24/03/2008 14:26 Re: builder_bigdisk_v3 now available [Re: mlord]
drakino
carpal tunnel

Registered: 08/06/1999
Posts: 7868
Originally Posted By: mlord
(We really need a <pre> tag!!):


I'm assuming here that you want the code to not be put into a 150px tall scrolling box. If thats correct, I've just modified it to no longer try to calculate the height and cap at 150. This isn't retroactive though, but future uses of the code block should work like it is in your post now.

Random note for any web developers here, the built in developer tools with Safari 3.1 made this really easy to track down. I haven't done much web debugging in any browser lately, but this feature really impressed me. All I did was right click the area I was interested in, and selected Inspect Element. I could then tweak the height tag to verify it was changing what I wanted. It was also displaying the CSS hierarchy to explain where all the attributes were coming from. Really wish tools like this existed years ago when I actually did web stuff for a living.


Edited by drakino (24/03/2008 14:28)

Top
#308503 - 24/03/2008 16:39 Re: builder_bigdisk_v3 now available [Re: drakino]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
*Much* better, thanks!

Top
#308508 - 24/03/2008 18:58 Re: builder_bigdisk_v3 now available [Re: drakino]
oliver
addict

Registered: 02/04/2002
Posts: 691
Originally Posted By: drakino
Random note for any web developers here, the built in developer tools with Safari 3.1 made this really easy to track down. I haven't done much web debugging in any browser lately, but this feature really impressed me. All I did was right click the area I was interested in, and selected Inspect Element. I could then tweak the height tag to verify it was changing what I wanted. It was also displaying the CSS hierarchy to explain where all the attributes were coming from. Really wish tools like this existed years ago when I actually did web stuff for a living.


Firebug for Firefox does the same thing, granted it isn't built into the browser like safari. It's also awesome for debugging async callback requests (ajax).
Lately I've been using it to optimize my websites, as I can see every single request and how long it took to download the requested file.
_________________________
Oliver mk1 30gb: 129 | mk2a 30gb: 040104126

Top
#308510 - 25/03/2008 03:25 Re: builder_bigdisk_v3 now available [Re: oliver]
Ross Wellington
enthusiast

Registered: 21/02/2006
Posts: 325
Hi,

I tried this and aside from a few minor difficulties, I was able to run the new Big Disk Builder v3 on a 250 GB Western Digital Parallel ATA drive.

A couple of problems I ran into the first time were, it completed the build normally (I think). I executed the Big Disk Builder v3 and then the car2_v2.01_hijack. Emplode (Windows XP Home) recognized the drive, showed available space of 232 GB. No hijack 488 loaded.

I loaded Hijack 488 onto it and then it wouldn't see the drive in emplode, though hijack worked.

Retried same procedure except this time it told me that the drives were already built. Same result.

I scribbled on the boot of the drive and retried the same procedure. Same result, good build, good player software, good hijack (button lights worked).

I uplugged the player, replugged it back in, good hijack v488. Retried emplode. This time it worked. Emplode saw the player, hijack buttons worked, verified vitals (drive shows up as a 244 GB though (not sure why - large format or reserves or what).

I am now loading a small set of 12 GB of music to it (mix of mostly .wav files and .mp3 files.

I'll let you know how it turns out.

But so far, it looks REAL GOOD!!

Thanks alot Mark, you are good!

Ross
_________________________
In SI, a little termination and attention to layout goes a long way. In EMC, without SI, you'll spend 80% of the effort on the last 3dB.

Top
#308511 - 25/03/2008 03:28 Re: builder_bigdisk_v3 now available [Re: Ross Wellington]
Ross Wellington
enthusiast

Registered: 21/02/2006
Posts: 325
Hi,

Forgot one thing, the disk builder FAQ link to the Big_Disk_Builder (Mark Lord's Special Disk Builder) is broken.

New file name....

Thanks again,

Ross
_________________________
In SI, a little termination and attention to layout goes a long way. In EMC, without SI, you'll spend 80% of the effort on the last 3dB.

Top
#308515 - 25/03/2008 10:08 Re: builder_bigdisk_v3 now available [Re: Ross Wellington]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31602
Loc: Seattle, WA
Fixed. Thanks.
_________________________
Tony Fabris

Top
#308517 - 25/03/2008 11:02 Re: builder_bigdisk_v3 now available [Re: Ross Wellington]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: Ross Wellington

I am now loading a small set of 12 GB of music to it (mix of mostly .wav files and .mp3 files.

I'll let you know how it turns out.
But so far, it looks REAL GOOD!!


I think we may still have a problem with it, so don't get too carried away just yet.

My theory is, that since the mkfs needed a *lot* of virtual memory (swap) before it would work, then the fsck will probably have the same issue.

Maybe not immediately, no, that would be too obvious. smile
But perhaps after you've painstakingly added 120GB of tracks to the beast -- *then* it might croak, with no easy means of recovery.

So another builder update will be needed before you can trust it on big drives. We need to change the partitioning scheme, as Roger suggested a while ago, to allocate a much larger swap partition on /dev/hda6.

The code that currently partitions the drive, is in a closed-source binary called init_drive, which is why I didn't fix it right off on the current pass through.

So I'll have to write a shell script to replace that binary, and then we can update the builder one more time. And I might even remember to co-locate the new swap (hda6) alongside the root (hda5) partition, in an attempt to minimize head movement when swapping.

And perhaps integrate the builder functionality into a new universal upgrade file, that includes the player software etc.. for a one-step install. But this would likely also require replacing the closed-source init binary with something more flexible.

And we should probably arrange for the swap to *always* be enabled whenever the player boots up, which will require another init change to force a mkswap on it each time around.

Cheers


Edited by mlord (25/03/2008 11:13)

Top
#308520 - 25/03/2008 12:16 Re: builder_bigdisk_v3 now available [Re: mlord]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5683
Loc: London, UK
Originally Posted By: mlord
But this would likely also require replacing the closed-source init binary with something more flexible.


Code:
while true; do /empeg/bin/player; /bin/bash; done


...is pretty all that init does. OK, OK that isn't the full story: it watches for particular exit codes from player to decide whether to start bash or not.

I wanted to extend it in v3 to map certain exit codes to different actions, and to put the desired exit code in the "Quit Player" protocol request packet (e.g. to run rsync in non-daemon mode and then restart the player when it was finished), but we never got around to it.

init is a binary because we didn't want the shell loaded when we didn't need it, and it's closed source because Hugo wrote it, and his code's always embarassing ;-)

No, seriously, it's closed source probably because it didn't seem worth the effort to open it.
_________________________
-- roger

Top
#308524 - 25/03/2008 13:13 Re: builder_bigdisk_v3 now available [Re: Roger]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: Roger
init is a binary because we didn't want the shell loaded when we didn't need it, and it's closed source because Hugo wrote it, and his code's always embarassing ;-)

No, seriously, it's closed source probably because it didn't seem worth the effort to open it.


Quite understandable, and not difficult to replace it either. smile

Thanks.

Top
#308525 - 25/03/2008 13:16 Re: builder_bigdisk_v3 now available [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
Code:
..
if [ -d /drive0/fids ]; then
        log "Drives already built"
else
..

Ahh.. I should also remember to have it still build DRIVE1 in that case, if necessary.

Top
#308526 - 25/03/2008 13:18 Re: builder_bigdisk_v3 now available [Re: Roger]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
It's also responsible for mounting the filesystems (the job done in full-size Linux by "mount -a"). The stuff about starting bash or not has to be right for synchronisation to work (synchronisation makes the player exit with a return code that causes init to respawn the player binary only, and not enter bash or reboot the unit). You can probably determine the value in question by running the player from the shell, then making it exit using "emptool --restart".

You also need to not be starting bash if it's not there (consumer image).

Peter

Top
#308527 - 25/03/2008 13:27 Re: builder_bigdisk_v3 now available [Re: peter]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: peter
It's also responsible for mounting the filesystems (the job done in full-size Linux by "mount -a"). The stuff about starting bash or not has to be right for synchronisation to work (synchronisation makes the player exit with a return code that causes init to respawn the player binary only, and not enter bash or reboot the unit). You can probably determine the value in question by running the player from the shell, then making it exit using "emptool --restart".

You also need to not be starting bash if it's not there (consumer image).

Peter


Thanks. For now, I'll just ignore it, and prepend an "init" script to run before it if I need anything special. The script would then exec the init binary after everything is set up.

But the partitioner is somewhat trickier to write anew, and higher on my list of Things That Ought To Get Done.

Cheers

Top
#308528 - 25/03/2008 13:29 Re: builder_bigdisk_v3 now available [Re: mlord]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
Originally Posted By: mlord
And we should probably arrange for the swap to *always* be enabled whenever the player boots up, which will require another init change to force a mkswap on it each time around.

I'm not so sure that's a good idea, at least for in-car use; I bet Linux starts using its swap, if available, at much lower memory pressures than are generated during normal car-player use. So the disk would keep getting spun up. Plus, I have a feeling the player issues a swapoff() during startup anyway, in case a previous synchronise died and left it enabled.

Peter

Top
#308530 - 25/03/2008 14:15 Re: builder_bigdisk_v3 now available [Re: peter]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: peter
I bet Linux starts using its swap, if available, at much lower memory pressures than are generated during normal car-player use. So the disk would keep getting spun up.

That's generally tunable to taste using /proc/sys/* interfaces, and I suppose we also happen to have the full source code if we care to get fancier.

Quote:
Plus, I have a feeling the player issues a swapoff() during startup anyway, in case a previous synchronise died and left it enabled.

That's easy enough to intercept. smile

Does the player normally try to enable swap during a sync?
Note that swapon -a uses /etc/fstab for the "-a" part, and if swap partitions are not listed there, it has no effect.

swapoff -a should work regardless, though.

Top
#308532 - 25/03/2008 16:21 Re: builder_bigdisk_v3 now available [Re: mlord]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
Originally Posted By: mlord
That's generally tunable to taste using /proc/sys/* interfaces, and I suppose we also happen to have the full source code if we care to get fancier.

Sure, I was just suggesting that most people's taste would be "not at all" during normal playback, which is more easily achieved with swapoff() than /proc/sys/*.

Quote:
Does the player normally try to enable swap during a sync?
Note that swapon -a uses /etc/fstab for the "-a" part, and if swap partitions are not listed there, it has no effect.

IIRC it's not enabled for the whole sync, but it's certainly enabled if fsck needs to be invoked. The player uses swapon(2) not swapon(8), so /etc/fstab isn't consulted, but of course it wouldn't be hard in Hijack to make each call to swapon(2)/swapoff(2) actually do more than one partition if that's what your new disk setup needed. I'm also pretty sure it uses "/swapfile", or some such name, as the argument to swapon(2), rather than the actual partition device, so you could point that symlink somewhere else if it helped.

Peter

Top
#308548 - 25/03/2008 23:25 Re: builder_bigdisk_v3 now available [Re: Roger]
altman
carpal tunnel

Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
Whilst I won't disagree that *some examples* of my code are nasty, I don't think init was my code smile

ISTR this was Mike's work. Lots of unixy-shelly things needed to be done there to make stuff work as expected. I remember ctrl-C not working in shells launched by it for a while!

Hugo

Top
#308550 - 25/03/2008 23:37 Re: builder_bigdisk_v3 now available [Re: mlord]
Ross Wellington
enthusiast

Registered: 21/02/2006
Posts: 325
Hi,


Mark said....
Maybe not immediately, no, that would be too obvious.
But perhaps after you've painstakingly added 120GB of tracks
to the beast -- *then* it might croak, with no easy means of
recovery.


Right again!

The player loaded 7 GB and then stopped with a load of warning meaasges. The warning messages had the track number, song title, artist, album as required and then an 8 character hexadecimal suffix. There was a warning file for each song not loaded. I re-sync'd for grins, and it sync'd fine with empty directories for the songs not loaded onto the drive.

Doesn't anything like The Beatles?

Ross
_________________________
In SI, a little termination and attention to layout goes a long way. In EMC, without SI, you'll spend 80% of the effort on the last 3dB.

Top
#308863 - 05/04/2008 12:12 Re: builder_bigdisk_v3 now available [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: Ross Wellington
Have you had time to work on the new builder for large drives?


Originally Posted By: mlord
the partitioner is somewhat trickier to write anew, and higher on my list of Things That Ought To Get Done.


Not yet.

Cheers

Top
#308865 - 05/04/2008 13:49 Re: builder_bigdisk_v3 now available [Re: mlord]
Ross Wellington
enthusiast

Registered: 21/02/2006
Posts: 325
Okay thanks>

Ross
_________________________
In SI, a little termination and attention to layout goes a long way. In EMC, without SI, you'll spend 80% of the effort on the last 3dB.

Top
#308866 - 05/04/2008 14:09 Re: builder_bigdisk_v3 now available [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
the partitioner is somewhat trickier to write anew, and higher on my list of Things That Ought To Get Done.


Okay, so I did some preliminary work on it this morning.
It's been about a decade since I last looked at low level partition info on IDE drives, so it took a while for it all to come back.

It's not rocket science, but it's not trivial either.


Basic partitioner strategy:

Use GETGEO or IDENTIFY to get drive C/H/S + LBA parameters
Do standard CHS geometry scaling to taste, and to comply with expectations for Large LBA geometries.


Heck, just get them from my /proc/ide/hd?/geometry entries.. duh!.

Construct partition tables as follows:

Extended partition /dev/hda1:
/dev/hda5: 16MB root (or maybe 32MB to give room for more apps)
/dev/hda6: 64MB swap

Other primary partitions:
/dev/hda2: 32MB spare (normally, nothing uses this space)
/dev/hda3: 16MB dynamic data (or maybe double it?)
/dev/hda4: many GB everything else

The current partitioner likes to round up partitions to even cylinders, but we really do not need to do this if we don't want to, apart from padding the MBR out to a full 63 sectors for universal compatibility.

Anyone have any comments or important things to point out or suggest?

In particular, would the player binary actually use a larger dynamic data area?? I know we could put longer bookmarks (and a longer current running order) in their if we left enough space, and we already know how to patch the player binary for those..

Cheers


Edited by mlord (05/04/2008 14:22)

Top
#308867 - 05/04/2008 14:14 Re: builder_bigdisk_v3 now available [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
As for implementation, there're a few methods I'm currently considering:
  • write a C-program to just do it all, or
  • use a shell script, with pipes into /bin/fdisk for the low level details.
  • stick /bin/gawk onto the image and use an awk program piped into /bin/fdisk for the low level details.

The first choice means I could take and incorporate it all inside the Hijack kernel, if that turned out to be useful.

The scripting choices may be more work, but are far less opaque in the end (easier to understand, test, and modify).

Cheers


Edited by mlord (05/04/2008 14:23)

Top
#308868 - 05/04/2008 17:13 Re: builder_bigdisk_v3 now available [Re: mlord]
Ross Wellington
enthusiast

Registered: 21/02/2006
Posts: 325
Hi,

I had a question in reference to the partitioning and caching of the current song. When a block of the current song is loaded, does any of it get loaded into a disk partition or does it go directly to the RAM for caching? Is this a PIO transfer or DMA?

The reason why I ask is I play mostly .wav files and have noticed that after it has played the first ~30 seconds to 1 minute of the song, there will be short break in the song during the next block fetch. After that, it doesn't have a break in the music (for up to hours). If a song is skipped forward or backward, it will do the same thing (as expected - It's doing a cache flush, right), and play the song for a short time have a short break, then resume the song(s).

A couple of other questions...

1) Will increasing the player RAM help expand the cache or is the cache a fixed size and the RAM would only expand the program space for 3rd party programs, display data, etc?

2) If the RAM was expanded for cache would it's performance be compromised due to a larger cache-flush requiring more time to do it? Will the break be more frequenct or a longer pause?

3) Is the cache 4 way set associative? If not, would that help the performance for the problem I mentioned above?

4) Does the player use the on-disk drive cache? Can it be utilized for cache queuing (sort of like L1, L2 cache so to speak)? Would that improve the performance if it was used?


The problem I mention above is only a minor annoyance. Mostly, I am really asking for an understanding of how things work, for general interest.

Is there a reference that defines the function of the partitions?

By expanding the partition sizes as you mentioned above, how would that improve performance?

Thanks again, and please excuse my ignorance on the operation of the player.

Ross
_________________________
In SI, a little termination and attention to layout goes a long way. In EMC, without SI, you'll spend 80% of the effort on the last 3dB.

Top
#308871 - 06/04/2008 00:59 Re: builder_bigdisk_v3 now available [Re: Ross Wellington]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: Ross Wellington
When a block of the current song is loaded, does any of it get loaded into a disk partition or does it go directly to the RAM for caching?

The player software loads it into a malloc'd buffer in RAM. Which means that the Linux kernel reads it into the page-cache (RAM), and them copies it to another location in RAM specified by the player software. Note to software developers: use mmap() instead of read() to avoid the memcpy and 2X memory requirement!!

Quote:
Is this a PIO transfer or DMA?

PIO. The empeg hardware completely lacks any provision for DMA from any device.

Quote:
1) Will increasing the player RAM help expand the cache or is the cache a fixed size and the RAM would only expand the program space for 3rd party programs, display data, etc?

2) If the RAM was expanded for cache would its performance be compromised due to a larger cache-flush requiring more time to do it? Will the break be more frequenct or a longer pause?

Since PIO is used (100% CPU during disk reads), and since the player software actively fights the Linux kernel's automatic caching, the result will be no different. Things will be jittery, possibly with a playback gap, while software is filling RAM with pre-read tracks. Since the RAM is larger, it will take longer to fill it. Where expanded RAM is useful, is in holding the larger data structures that come with large drives -- the filesystem metadata, and the empeg player's database.

Quote:
3) Is the cache 4 way set associative? If not, would that help the performance for the problem I mentioned above?

Dunno to the first question, and not likely to the second. X-way set associative memory caches only help with data that's already in RAM. The problem in the empeg is that there's not much RAM, and the data is on disk to begin with.

Quote:
4) Does the player use the on-disk drive cache?

The player is unaware of it, and the drive automatically uses it.

Quote:
Can it be utilized for cache queuing (sort of like L1, L2 cache so to speak)? Would that improve the performance if it was used?

Already happening. You already see the results.

Quote:
Is there a reference that defines the function of the partitions?

I think it's probably somewhere in the empeg FAQ.

Quote:
By expanding the partition sizes as you mentioned above, how would that improve performance?

Not. But a large drive is more likely (than a small drive) to have large numbers of tracks, possibly exceeding the design limits of the player software. One such group of limits has to do with the maximum number of FIDs ("tracks") in the current running order (or in a bookmark list), and how well this playlist is remembered across power cycles.

The memory for the current running order (and bookmarks) is actually the dynamic data partition on disk. Making this partition larger, in conjunction with some software changes (eg. a modified version of the set_empeg_max_fid hack), could alleviate those particular limits.

* * *

EDIT: A somewhat related discussion came up on the Linux Kernel Mailing List (LKML) last week. I had raised some issues there regarding the 2TB drives that are soon to hit the market, and the ensuing discussion eventually touched upon the same immediate issue that you have experienced: memory requirements for filesystem checks (fsck).

Ted (Tso) suggested that, for a 2TB filesystem, the machine might need as much as 2GB of memory for fsck to be guaranteed to run without crashing..


Cheers


Edited by mlord (06/04/2008 01:04)

Top
#308874 - 06/04/2008 12:57 Re: builder_bigdisk_v3 now available [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Here is my first crack at a script to reinitialize an empeg partition table, using larger sizes for the root, swap, and dynamic data partitions.

After comments here, I'll integrate it into a new set of bigdisk_builder images.

Please have a look, and let me know if the sizes look good, and if you can find/fix any potential bugs.

In particular, you can install this script onto an empeg, and run it -- by default it will not actually commit changes to disk, so it is safe to play with.

EDIT: updated again. Current version sticks with an old-fashioned 16MB root filesystem instead of something larger.

Thanks
Code:
#!/bin/bash

function del_partitions(){
        echo "d"
        echo "1"
        echo "d"
        echo "2"
        echo "d"
        echo "3"
        echo "d"
        [ "$SPECIAL4" = "4" ] || echo "4"
}

function enable_errcheck(){
        echo "v"
}

function add_partition(){
        echo "n"
        echo "$1"
        [ "$1" != "l" -a "$2" != "$SPECIAL4" ] && echo "$2"
        echo
        echo "$3"
        if [ "$4" != "" ]; then
                echo "t"
                echo "$2"
                echo "$4"
        fi
}

function make_partitions(){
        del_partitions
        enable_errcheck
        [ "$UNITS" = "K" ] && echo "u"
        add_partition e 1 "+$SZ_HDA1$UNITS"
        add_partition l 5 "+$SZ_HDA5$UNITS" 83
        add_partition l 6 ""                82
        add_partition p 2 "+$SZ_HDA2$UNITS" 83
        add_partition p 3 "+$SZ_HDA3$UNITS" 10
        add_partition p 4 ""                83
        echo "a"
        echo "5"
        echo "p"
        echo $COMMIT
}

function errcheck(){
        DOECHO=0
        ERRCHECK=0
        while read x ; do
                [[ "$x" == Disk\ ${DEV}:*             ]] && DOECHO=1
                [[ "$x" == Command\ \(m\ for\ help\): ]] && DOECHO=0
                [ $DOECHO -eq 1 ] && [[ "$x" != *does\ not\ end\ on* ]] && echo "$x" >&2
                [[ "$x" == *\ unallocated\ sectors    ]] && ERRCHECK=1
                if [ $ERRCHECK -eq 1 ]; then
                        if [[ "$x" == *out\ of\ range.          \
                           || "$x" == *got\ EOF\ thrice*        \
                           || "$x" == *No\ free\ sectors*       \
                           || "$x" == *unknown\ command*        \
                           ]]
                        then
                                echo "ERROR: fdisk: $x" 1>&2
                                exit 1
                        fi
                fi
        done
        exit 0
}

## First parameter is the device path, eg. /dev/hda:
[ "$1" = "" ] && echo "missing device path" && exit 1
DEV=$1

## The default is to "just pretend", and not actually do anything for real.
## Supply "COMMIT" on the command line to actually write the new partition info.
COMMIT=q
[ "$2" = "COMMIT" ] && COMMIT=w

## Figure out how big $DEV is, in kilobytes:
DEV_KB=`sed -n -e"s/  *[0-9][0-9]*  *[0-9][0-9]* *\([0-9]*\) ${DEV##*/}\$/\1/p" < /proc/partitions`
if [ "$DEV_KB" = "" ]; then
        echo "$DEV: not found"
        exit 1
fi

## Newer fdisk versions don't always prompt for partition number 4, but the empeg does:
SPECIAL4="4"
[ "`fdisk -v`" = "fdisk v2.10d" ] && SPECIAL4=NONE      ## empeg

## minimum partition sizes, in $UNITS (defined below):
SZ_HDA5=16      ## root
SZ_HDA6=64      ## swap
SZ_HDA1=$(( SZ_HDA5 + SZ_HDA6 ))
SZ_HDA2=32      ## spare
SZ_HDA3=32      ## dynamic data
SZ_HDA4=        ## FIDs, use all remaining space

## allocation units, normally "M" (megabytes), or possibly "K" (kilobytes):
UNITS=M

## "K" kilobytes: only for testing here on a tiny drive I have:
SZ_TOTAL=$(( SZ_HDA1 + SZ_HDA2 + SZ_HDA3 + 128 ))
[ $(( SZ_TOTAL * 1024 )) -ge $DEV_KB ] && UNITS=K

echo
[ "$SPECIAL4" = "4" ] || echo "$DEV: ${DEV_KB} KB"
make_partitions $COMMIT | fdisk $DEV 2>&1 | errcheck || exit 1
echo "Success"


Edited by mlord (06/04/2008 16:46)

Top
#308875 - 06/04/2008 14:12 Re: builder_bigdisk_v3 now available [Re: mlord]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
Originally Posted By: mlord

/dev/hda5: 16MB root (or maybe 32MB to give room for more apps)

How well does that work with stock upgrade files? Currently, owners of (already built) big drives can apply any stock upgrade, and then immediately a new Hijack (only), and be fine. IIRC the stock car-player upgrade files contain complete root filesystem images, not tarballs (we only invented embedded tarballs for Jupiter). So they'll come out as 16M, not 32M, filesystems.

Mind you, I suppose anyone installing a stock upgrade is going to eliminate any "more apps" from their root filesystem anyway. I think the recommendation has to remain that "more apps" live on hda2, hda4, or hdb5.

Quote:
/dev/hda6: 64MB swap

That's definitely a good idea, though.

Quote:
In particular, would the player binary actually use a larger dynamic data area?? I know we could put longer bookmarks (and a longer current running order) in their if we left enough space, and we already know how to patch the player binary for those..

The table of offset/length pairs which set-empeg-max-fid already patches, is the only thing which tells the player how big a dynamic data area it has. So no, it won't use a bigger area automatically, but you already know how to tell it to use a bigger area. Nor (if building a new drive from empty) is it important to keep the offsets/lengths compatible with the stock player -- as long as users know they can then never use a stock player on that drive.

Peter

Top
#308881 - 06/04/2008 16:37 Re: builder_bigdisk_v3 now available [Re: peter]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: peter
Originally Posted By: mlord

/dev/hda5: 16MB root (or maybe 32MB to give room for more apps)

How well does that work with stock upgrade files?

It works fine -- the extra space is simply ignored, and one ends up with a 16MB root filesystem, with 16MB of unused spaced at the end of it, just as already happens on some big drives today.

Quote:
Currently, owners of (already built) big drives can apply any stock upgrade, and then immediately a new Hijack (only), and be fine.

Mmm.. I was going to say, no that's actually unsafe to do, because if the player ever runs (even once), it will write to the disk (dynamic data), and possibly screw up (write to the wrong part of the disk) due to lack of working LBA48 support in the kernel.

But in practice, the entire dynamic data partition is still well below the LBA28 boundary, so it actually will work and is safe to do -- just never try to remount the music partitions r/w.

In practice, though, there is no reason to ever use a "stock" .upgrade file on a bigdisk player, as there are hijacked versions available to use instead.

But it could indeed get a bit messy, because if we go ahead with 32MB root on bigdisks, then we'll have a mix of 16MB and 32MB players out there, and the 32MB .upgrade files won't work on the 16MB players (or worse, the firmware might happily write out all 32MB..).

So, I guess the solution for space on root has to be to use the spare partition, and perhaps update /bin/init etc.. to automatically mount it at boot, and have rw/rwm mount it r/w along with the others.

Ugh. I guess I'll first check and see what the firmware does with a 32MB root file on a 16MB partition..

Cheers


Edited by mlord (06/04/2008 16:42)

Top
#308882 - 06/04/2008 16:53 Re: builder_bigdisk_v3 now available [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Here is what happens now on a 20GB disk I have here.

First, the "before" view, from a stock .upgrade file:
Code:
Disk /dev/hda: 16 heads, 63 sectors, 38760 cylinders
Units = cylinders of 1008 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1             1        66     33232+   5  Extended
/dev/hda2            67       132     33264   83  Linux
/dev/hda3           133       165     16632   10  OPUS
/dev/hda4           166     38760  19451880   83  Linux
/dev/hda5             1        33     16569   83  Linux
/dev/hda6            34        66     16600+  82  Linux swap


Next, the "after" view, from running the new partitioning script:
Code:
Disk /dev/hda: 16 heads, 63 sectors, 38760 cylinders
Units = cylinders of 1008 * 512 bytes

Device Boot    Start       End    Blocks   Id  System
/dev/hda1             1       163     82120+   5  Extended
/dev/hda2           164       229     33264   83  Linux
/dev/hda3           230       295     33264   10  OPUS
/dev/hda4           296     38760  19386360   83  Linux
/dev/hda5   *         1        33     16569   83  Linux
/dev/hda6            34       163     65488+  82  Linux swap


Of course, the idea is not to run this on an empeg that has already been set-up, but I just wanted to show what would be done differently.

Mmm.. since the root filesystem (/dev/hda5) is now stuck at 16MB, perhaps we should increase the size of the spare (/dev/hda2) partition?

Top
#308884 - 06/04/2008 17:00 Re: builder_bigdisk_v3 now available [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Ross:

Could you perhaps install that script onto one of your already-built 250GB drives, and run it and post the results here?

Eg. Save the script as /reinit_partition_table.sh, and upload to the player via FTP:

ftp your.players.ip.addr
(login as root, no passwd)

site rw
put reinit_partition_table.sh
chmod 755 reinit_partition_table.sh
site ro
quit


Then connect over serial, hit Control^C to get a shell prompt, and run the script:

/reinit_partition_table.sh /dev/hda


It will pretend to repartition the drive, and it will print out the results of what would have been. But it won't actually hurt anything for now.

I just want to see if the copy of fdisk on it correctly handles the 250GB drive.

Thanks


Attachments
reinit_partition_table.sh (456 downloads)
Description: reinit_partition_table.sh




Edited by mlord (06/04/2008 17:07)

Top
#308885 - 06/04/2008 17:13 Re: builder_bigdisk_v3 now available [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
Ross:

Could you perhaps install that script onto one of your already-built 250GB drives, and run it and post the results here?


And if you want to help things along even more, you could then have it make the changes permanent, by adding the word COMMIT to the end of the command line when running it, as in:
Code:
/reinit_partition_table.sh /dev/hda COMMIT

This will destroy any music on the machine, and you must then re-install the player .upgrade image again after running it.

Give that a try, add some tunes, add some more tunes, and generally beat on things. If it all seems to be working, then I'll roll it into a bigdisk_builder, and respin the .upgrade files etc..

Actually, I might eliminate the need for the builder entirely.. perhaps having Hijack automatically create a partition table when it sees a blank drive. Except for the music partition (/dev/hda5). Then the player software .upgrade files could format the swap on boot, and create/format /dev/hda5 to fit.

That might be nice. I wonder what I'm forgetting here?


Edited by mlord (06/04/2008 17:14)

Top
#308889 - 06/04/2008 17:50 Re: builder_bigdisk_v3 now available [Re: mlord]
Roger
carpal tunnel

Registered: 18/01/2000
Posts: 5683
Loc: London, UK
Originally Posted By: mlord
That might be nice. I wonder what I'm forgetting here?


When the drive's not blank, but you want to rebuild it anyway? ...

Yeah, I know that you can just cat </dev/zero >/dev/hda for a bit, but...
_________________________
-- roger

Top
Page 2 of 8 < 1 2 3 4 5 6 7 8 >