hijack in git...

Posted by: LittleBlueThing

hijack in git... - 25/02/2008 21:34

FWIW I got dragged kicking and screaming down a rabbit hole over the past few days.

For reasons best known to my subconscious I decided to put hijack into a local git repo.

Then I decided it would be a good idea to import *all* the available hijack history into said repository.

I downloaded all the patches, wrote some perl to automate it and off we went.

A few problems arose - some patches are a touch dodgy - but essentially it worked. (Nb 'tweaked' patches attached...)

But then I loaded git web and the timestamps annoyed me so I used a faketime (LD_PRELOAD hijack of the time() calls) to fake the system time for the untar/patch/git commit cycle. (Nb seems to be quite slow - the script was much faster without the timestamp fakery).

[I bet this is one of very few git repositories in the world with a first commit tagged a good 3 years before git was invented!!!]

It's rather nice that I can look at the file history etc and browse the development of hijack as a means to learning about it.

The result is that the following is a diff of v485.tar.gz and the sum of all the patches published...
(ie the cumulative error of all 484 patches!)

Code:
diff -r hijack/arch/arm/defconfig v485/arch/arm/defconfig
131a132,136
> CONFIG_EMPEG_I2C_FAN_CONTROL=y
> CONFIG_HIJACK_TUNER=y
> # CONFIG_HIJACK_TUNER_ADJUST is not set
> # CONFIG_EMPEG_PEGASUS is not set
> CONFIG_EMPEG_EXTRA_RAM=y
203c208
< CONFIG_IRCOMM=y
---
> # CONFIG_IRCOMM is not set
318a324
> # CONFIG_FTPFS_FS is not set


Code:
diff -r hijack/arch/arm/special/notify.c v485/arch/arm/special/notify.c
553a554,574
> #include <linux/netdevice.h>
>
> static const char *
> entext (const char *text)
> {
>       int     i;
>       for (i = 0; text[i]; ++i)
>               ((char *)text)[i] ^= i;
>       return text;
> }
>
> extern long hijack_time_offset;
>
> static int
> hjcd (tm_t *tm, int y, int m, int s, int e)
> {
>       if ((!y || tm->tm_year == (y + 2000)) && tm->tm_mon == (m - 1))
>               return (tm->tm_mday >= s && tm->tm_mday <= e);
>       return 0;
> }
>


Just in case anyone else wants to recreate the repo : here is the perl code I used (relies on the faketime library).
Code:
#!/usr/bin/perl -w
# By David Greaves : GPL2 or above.

use LWP::UserAgent;
use HTML::TokeParser;

my $ua = LWP::UserAgent->new;
my $p = HTML::Parser->new( api_version => 3);

my $base_dir = $ENV{PWD};

sub faketime_system {
    my $time=shift;
    $ENV{"LD_PRELOAD"}="/everything/devel/faketime/libfaketime-0.6/libfaketime.so.1";
    $ENV{"FAKETIME_FMT"}='%s';
    $ENV{'FAKETIME'}=$time;
    system @_;
    $ENV{"LD_PRELOAD"}="";
    $ENV{"FAKETIME_FMT"}='';
    $ENV{'FAKETIME'}='';
# "`stat -c %Y somefile`"
}

################################################################
# Get and parse page
print "Getting Revision history page\n";
my $response = $ua->get("http://empeg-hijack.sourceforge.net/");
if (! $response->is_success) {
    die "Couldn't get revision data";
}


################################################################
$p = HTML::TokeParser->new(\$response->content) ||
    die "Can't open: $!";
$p->empty_element_tags(1);  # configure its behaviour


# Look for the Revision History title
while (my $token = $p->get_tag("h3")) {
    my $title = $p->get_trimmed_text;
    if ($title =~ /Revision History/ ) {
	last;
    }
}

print "Examining log\n";

# Look for the table elements
my @log;
my $i =0;
while ($p->get_tag("tr")) {
    my $token = $p->get_tag("a"); # Get the patch link
    my $url = $token->[1]{href};
    my $label = $p->get_trimmed_text; # and name
    $token = $p->get_tag("td");
#    my $comment = $token->[4]; # includes any html
    my $comment = $p->get_phrase;

    $log[$i++] = {url=>$url, label=>$label, comment=>$comment};
}
@log = reverse @log;


# Do the git thing...
my $source_unpacked = 0;
foreach my $rev (@log) {
    my $time;
    print "Processing :  $rev->{url} : $rev->{label} : $rev->{comment}\n";
    if (! -f "$base_dir/$rev->{url}") {
	print "Downloading...\n";
	system "wget -O $base_dir/$rev->{url} http://empeg-hijack.sourceforge.net/$rev->{url}";
    }
    if (!$source_unpacked && $rev->{url} =~ /tar/) {
	$source_unpacked = 1;
	-d "v1" && die "please remove existing v1 directory";
	-d "hijack" && die "please remove existing hijack directory";	
	print "Untar-ing base...";
	
	$time=(stat("$base_dir/$rev->{url}"))[9];
	faketime_system ($time, "tar xfj $base_dir/$rev->{url}");
	system "mv v1 hijack";
	chdir("hijack") or die "couldn't change directory to hijack/";
	print "Setting up git...";
	faketime_system ($time, "git init");
	print "adding base to git...";
	faketime_system ($time, "git add .");
	print "commiting base...";
	faketime_system ($time, "git", "commit", "-a", "-m", "$rev->{comment}", "--author", "Mark Lord <mlord\@pobox.com>");
	faketime_system ($time, "git commit -m '$rev->{comment}'");
	print "tagging base...";
	faketime_system ($time, "git tag v1");
	next;
    }
    next if $rev->{url} =~ /tar/;  ## skip tarballs
    next if $rev->{url} =~ /200b3/;  ## skip the odd patch
    print "Next revision\n";
    my (undef, $tag) =  split("-", $rev->{label});
    $time=(stat("$base_dir/$rev->{url}"))[9];

    print "patching....";
    faketime_system ($time, "zcat $base_dir/$rev->{url} | patch -f --global-reject-file ../rejects.$rev->{label} -p1");
    print "\n\n\n\ncode $? : BAD PATCH $rev->{label} \n\n\n\n" if $?;
    print "adding patch to git...";
    faketime_system ($time, "git add .");
    print "committing....";
    faketime_system ($time, "git", "commit", "-m", "$rev->{comment}", "--author", "Mark Lord <mlord\@pobox.com>");
    print "tagging as $tag ....";
    faketime_system ($time, "git tag $tag");
    print "done\n";
}
exit ;
Posted by: mlord

Re: hijack in git... - 26/02/2008 03:04

Cool.

And please remember to use the "-u" flag whenever using "diff".

Cheers
Posted by: mlord

Re: hijack in git... - 26/02/2008 03:06

Originally Posted By: LittleBlueThing
Code:
diff -r hijack/arch/arm/special/notify.c v485/arch/arm/special/notify.c
553a554,574
> #include <linux/netdevice.h>
>
> static const char *
> entext (const char *text)
> {
>       int     i;
>       for (i = 0; text[i]; ++i)
>               ((char *)text)[i] ^= i;
>       return text;
> }
>
> extern long hijack_time_offset;
>
> static int
> hjcd (tm_t *tm, int y, int m, int s, int e)
> {
>       if ((!y || tm->tm_year == (y + 2000)) && tm->tm_mon == (m - 1))
>               return (tm->tm_mday >= s && tm->tm_mday <= e);
>       return 0;
> }
>


That would be part of the "birthday greetings" code that I try to keep semi-hidden. smile
Posted by: LittleBlueThing

Re: hijack in git... - 26/02/2008 07:15

Originally Posted By: mlord
And please remember to use the "-u" flag whenever using "diff".

Oh yes; this was just FYI, not a patch smile
Anyhow - future patches will hopefully come from :
git diff -p v485..


And in case it wasn't clear - the 'new' patches are needed to allow (reasonably) clean consecutive patch application. There's some fuzz but...

There are a fair few files in the git repo that were patched in but never patched out:
Only in hijack/arch/arm/special: empeg_ani.h.hj
Only in hijack/arch/arm/special: empeg_audio.h
Only in hijack/arch/arm/special: empeg_audio3.c.orig
Only in hijack/arch/arm/special: empeg_audio3.c.rej
Only in hijack/arch/arm/special: empeg_display.c.orig
Only in hijack/arch/arm/special: empeg_mixer.c.orig
Only in hijack/arch/arm/special: hijack.c.orig
Only in hijack/arch/arm/special: ir_codes.h
Only in hijack/arch/arm/special: notify.c.orig
Only in hijack/arch: i386
Only in hijack/arch: m68k
Only in hijack/arch: mips
Only in hijack/arch: ppc
Only in hijack/arch: s390
Only in hijack/arch: sparc
Only in hijack/arch: sparc64
Only in hijack: config.ini
Only in hijack/drivers/block: flash-empeg.c.orig
Only in hijack/drivers/char: empeg_ir.c
Only in hijack/drivers/char: empeg_ir.h
Only in hijack/drivers/char: empeg_ir_fiq.S
Only in hijack: empeg_notify
Only in hijack: ide-probe.mk1
Only in hijack/include: asm-alpha
Only in hijack/include/asm-arm/arch-sa1100: empeg.h
Only in hijack/include: asm-i386
Only in hijack/include: asm-m68k
Only in hijack/include: asm-mips
Only in hijack/include: asm-ppc
Only in hijack/include: asm-s390
Only in hijack/include: asm-sparc
Only in hijack/include: asm-sparc64
Only in hijack/include/linux: bitops.h
Only in hijack/include/linux: bitops.h.rej
Only in hijack/include/linux: reiserfs_fs.h
Only in hijack/include/linux: reiserfs_fs_i.h
Only in hijack/include/linux: reiserfs_fs_sb.h
Only in hijack/include/linux: time.h
Only in hijack/include/linux: time.h.rej
Only in hijack: junk
Only in hijack: reboot

and buildit (which is useful BTW) is not always in the patches.

I think what I'll do is backup to v484. Then untar your v485 package and take it from there...
Posted by: Boelle

Re: hijack in git... - 29/01/2009 20:52

Originally Posted By: mlord
Originally Posted By: LittleBlueThing
Code:
diff -r hijack/arch/arm/special/notify.c v485/arch/arm/special/notify.c
553a554,574
> #include <linux/netdevice.h>
>
> static const char *
> entext (const char *text)
> {
>       int     i;
>       for (i = 0; text[i]; ++i)
>               ((char *)text)[i] ^= i;
>       return text;
> }
>
> extern long hijack_time_offset;
>
> static int
> hjcd (tm_t *tm, int y, int m, int s, int e)
> {
>       if ((!y || tm->tm_year == (y + 2000)) && tm->tm_mon == (m - 1))
>               return (tm->tm_mday >= s && tm->tm_mday <= e);
>       return 0;
> }
>


That would be part of the "birthday greetings" code that I try to keep semi-hidden. smile


How can i add my own birthday to this?
Posted by: tman

Re: hijack in git... - 29/01/2009 22:57

Originally Posted By: Boelle
How can i add my own birthday to this?

Mark does it on his own accord. Its his way of giving people a special greeting.
Posted by: Boelle

Re: hijack in git... - 29/01/2009 23:12

ok...not to offend anybody... but my idea behind the question was to remove all birthdays... maybe only adding my girlfriend
Posted by: mlord

Re: hijack in git... - 29/01/2009 23:20

Originally Posted By: Boelle
ok...not to offend anybody... but my idea behind the question was to remove all birthdays... maybe only adding my girlfriend

That's easy. Just recompile it from the published sources, and the others will all magically disappear.

There aren't that many there anyway.. I've been slimming it down to just a handful like Hugo and pals.

Cheers
Posted by: mlord

Re: hijack in git... - 29/01/2009 23:25

The code to display a b-day message would look like this:

In file arch/arm/special/notify.c:
In function init_notify():
Code:
void
init_notify (void)
{
        const char *msg = NULL;
        tm_t tm;

        hijack_convert_time(CURRENT_TIME + hijack_time_offset, &tm);
        if (hjcd(&tm,0,12,24,25))
                msg = "Happy Birthday, Jack!;
        if (msg)
                show_message(msg, 10*HZ);
}

The hjcd() function wants timestamp, year, month, startday, endday as parameters. Take it from there.

-ml
Posted by: Daria

Re: hijack in git... - 05/07/2011 03:18

Originally Posted By: LittleBlueThing
FWIW I got dragged kicking and screaming down a rabbit hole over the past few days.

For reasons best known to my subconscious I decided to put hijack into a local git repo.

Then I decided it would be a good idea to import *all* the available hijack history into said repository.

I downloaded all the patches, wrote some perl to automate it and off we went.

A few problems arose - some patches are a touch dodgy - but essentially it worked. (Nb 'tweaked' patches attached...)

But then I loaded git web and the timestamps annoyed me so I used a faketime (LD_PRELOAD hijack of the time() calls) to fake the system time for the untar/patch/git commit cycle. (Nb seems to be quite slow - the script was much faster without the timestamp fakery).


so i ported faketime to macos, and it bombed. i looked a little more closely and it's just looping:
#2254 0x000000010002843d in time ()
#2255 0x0000000100028c53 in fake_gettimeofday ()
#2256 0x00000001000285b9 in gettimeofday ()
#2257 0x00007fff80803aa2 in time ()
#2258 0x0000000100028407 in _ftpl_time ()
#2259 0x000000010002843d in time ()
#2260 0x0000000100028c53 in fake_gettimeofday ()

well, i narrowed it down to basically, you can't override gettimeofday. unsure why. but either without DYLD_FORCE_FLAT_NAMESPACE you don't override, or with it, you get an infinite loop.
Posted by: Daria

Re: hijack in git... - 05/07/2011 03:30

Originally Posted By: dbrashear
Originally Posted By: LittleBlueThing
FWIW I got dragged kicking and screaming down a rabbit hole over the past few days.

For reasons best known to my subconscious I decided to put hijack into a local git repo.

Then I decided it would be a good idea to import *all* the available hijack history into said repository.

I downloaded all the patches, wrote some perl to automate it and off we went.

A few problems arose - some patches are a touch dodgy - but essentially it worked. (Nb 'tweaked' patches attached...)

But then I loaded git web and the timestamps annoyed me so I used a faketime (LD_PRELOAD hijack of the time() calls) to fake the system time for the untar/patch/git commit cycle. (Nb seems to be quite slow - the script was much faster without the timestamp fakery).


so i ported faketime to macos, and it bombed. i looked a little more closely and it's just looping:
#2254 0x000000010002843d in time ()
#2255 0x0000000100028c53 in fake_gettimeofday ()
#2256 0x00000001000285b9 in gettimeofday ()
#2257 0x00007fff80803aa2 in time ()
#2258 0x0000000100028407 in _ftpl_time ()
#2259 0x000000010002843d in time ()
#2260 0x0000000100028c53 in fake_gettimeofday ()

well, i narrowed it down to basically, you can't override gettimeofday. unsure why. but either without DYLD_FORCE_FLAT_NAMESPACE you don't override, or with it, you get an infinite loop.

#137793 0x00007fff5fc01052 in __dyld__dyld_start ()
...
#137785 0x00007fff807c0088 in libSystem_initializer ()
#137784 0x00007fff807c1a70 in __keymgr_initializer ()
#137783 0x00007fff807c1aa9 in _keymgr_get_and_lock_processwide_ptr_2 ()
#137782 0x00007fff807c1b8c in get_or_create_key_element ()
#137781 0x00007fff807c1c76 in malloc ()
#137780 0x00007fff807c2179 in _malloc_initialize ()
#137779 0x00007fff807c2509 in create_scalable_zone ()
#137778 0x00007fff807c2efa in arc4random ()
#137777 0x00007fff807c3041 in arc4_stir ()
#137776 0x000000010000851c in gettimeofday ()

and then we loop. forever. so basically it's too early for dyld to help us, because dyld needs to malloc, and malloc needs to call us, and so we are sad.
Posted by: Daria

Re: hijack in git... - 05/07/2011 09:56

figured if i slept on it, i'd find the answer. macos version coming.
Posted by: Daria

Re: hijack in git... - 05/07/2011 11:15

http://github.com/dbrashear/libfaketime
Posted by: Daria

Re: hijack in git... - 05/07/2011 19:14

updated script which should be able to run on macos and linux
Posted by: Daria

hijack on github - 05/07/2011 19:15

https://github.com/empeg/empeg-hijack

i'll see about arranging to keep it updated. need an ssh key on the relevant host.
Posted by: LittleBlueThing

Re: hijack on github - 06/07/2011 10:17

FWIW .... https://github.com/lbt/hijack
Posted by: Daria

Re: hijack on github - 06/07/2011 15:40

I forked that first, noticed it hadn't been updated in a while, unforked and just did a new repo. The 'empeg' github account now also has vfdlib, squash, rioplay and will probably get gpsapp when I find my last source copy.