Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 2 of 2 < 1 2
Topic Options
#37009 - 25/08/2001 09:06 Re: OT: Drive Image 4.0 [Re: drakino]
synergy
enthusiast

Registered: 20/02/2001
Posts: 345
Oh, and most of the slowdown is caused by another side effect of the worm, all the ARP requests it's generating. Since @Home neighboorhoods act like a LAN, my cable modem gets all the random ARP requests. The worm it's self is just proving what MCSE sysadmins out there have no clue, and how many @home users are violating their terms of service (in more ways then just having a web server now.)


I had forgetten completely about that... The ARP table. Chances are, your 2600 doesn't have a memory upgrade, so it's very likely that the arp table is filling up, causing the router to go into brainlock trying to deal with them all... That would explain why the system progressively gets worse after a reboot. A memory upgrade would help extend the live time, but honestly, the better solution is for the ISP to step in.

MCSE? My Computer Sucks Ethernet? That was what you were referring to, right?

_________________________
Synergy [orange]mk2, 42G: [blue] mk2a, 10G[/blue][/green] I tried Patience, but it took too long.

Top
#37010 - 25/08/2001 13:25 Re: OT: Drive Image 4.0 [Re: synergy]
drakino
carpal tunnel

Registered: 08/06/1999
Posts: 7868
I tend to think MCSE is a Minesweeper Consultant and Solitaire Expert

Thats a good one as well, haven't come across it before.


Top
#37011 - 25/08/2001 14:28 Re: OT: Drive Image 4.0 [Re: drakino]
synergy
enthusiast

Registered: 20/02/2001
Posts: 345
I tend to think MCSE is a Minesweeper Consultant and Solitaire Expert Thats a good one as well, haven't come across it before.




It seemed appropriate given the current situations....

_________________________
Synergy [orange]mk2, 42G: [blue] mk2a, 10G[/blue][/green] I tried Patience, but it took too long.

Top
#37012 - 25/08/2001 17:58 Re: OT: Drive Image 4.0 [Re: synergy]
jimhogan
carpal tunnel

Registered: 06/10/1999
Posts: 2591
Loc: Seattle, WA, U.S.A.
Synergy: "I had forgetten completely about that... The ARP table. Chances are, your 2600 doesn't have a memory upgrade, so it's very likely that the arp table is filling up, causing the router to go into brainlock trying to deal with them all... That would explain why the system progressively gets worse after a reboot."

I wondered about that -- ARP cache growth -- but couldn't see why it would overwhelm even an anemic 2600 unless Code Red was generating a broadcast in its random scan (maybe it is) or unless @HOME is supernetted such that it has huge numbers of hosts on each customer net (that may be the case for all I know).

Anyhow, I looked again and found CIAC/Cisco advisories that smell very much like what Doug describes. The CIAC advisory:

"The nature of the "Code Red" worm's scan of random IP addresses and the resulting sharp increase in network traffic can noticeably affect Cisco routers running Cisco IOS software, depending on the device, its current configuration, and the topology of the network. Unusually high CPU utilization and memory starvation may occur, and it can be mitigated in many cases simply by refining the configuration. Troubleshooting and configuration
recommendations are available at this location: http://www.cisco.com/warp/public/63/ts_codred_worm.shtml"

"high CPU utilization and memory starvation" -- perfect. Still I said ARP? broadcasts? So looked at that Cisco URL? In fact one explanation is along the lines of:

"Reducing ARP Input Memory Usage...
A huge memory usage in ARP Input occurs when there is a static route pointing to a broadcast interface, such as the following: ip route 0.0.0.0 0.0.0.0 Vlan3

Every packet for the default route will be sent to the VLAN3, but since there is no next hop IP address specified, the router will send an ARP request for the destination IP address, and the next hop router for that destination will reply with its own MAC address, unless proxy ARP is disabled. This creates an additional entry in the ARP table where the destination IP address of the packet will be mapped to the next-hop MAC address. Since the "Code Red" worm sends packets to random IP addresses, this adds a new ARP entry for each random destination address and consumes more and more memory under the ARP Input process."

Behaviorally, this maps pretty well to what Doug describes, but just not sure why an admin on an average network would map static route to broadcast as described. Anyhow, the tech note on that URL lists a whole host of other things that his network person should check out including a bug in an earlier revision of 26xx NAT code.

As above. See below. Will follow with interest. *

Jim

* The shortest possible chart entry that'll qualify a consulting doc to collect their fee! (Not fashioning myself a consulting specialist, but I just love that line!)

_________________________
Jim


'Tis the exceptional fellow who lies awake at night thinking of his successes.

Top
#37013 - 27/08/2001 01:15 Re: OT: Drive Image 4.0 [Re: jimhogan]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5539
Loc: Ajijic, Mexico
(have you measured this empirically?)

I haven't actually put a stop watch on it... but that really isn't necessary. When the router is freshly rebooted, it takes between 1--2 seconds to display the next post in a thread on this bbs. By the time 60--90 minutes have passed, it will take between 1--2 minutes to display the next post. Finally, Netscape crashes saying the server is off line, and I can no longer connect to any website until the router is rebooted.

I wish I were technically knowledgeable enough to respond to the rest of the things you said... but I have printed it out and will give it to my engineer tomorrow to see what he thinks.

Thanks...

tanstaafl.

"There Ain't No Such Thing As A Free Lunch"
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#37014 - 27/08/2001 02:39 Re: OT: Drive Image 4.0 [Re: tanstaafl.]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5539
Loc: Ajijic, Mexico
I guess the test will be to go to DOS mode and as an experiment use XCOPY /S/E/ to copy the drive and see how long that takes, just as a
comparison.


Well, XCOPY didn't work -- it cratered every time it came to a read-only, system, or hidden file.

However, my favorite disk-management program, Z-Tree for Windows, has a "Mirror" function that copies all files including subdirectories from one directory to another, or one partition to another, or one disk to another -- whatever. It only choked on a few files - like the AppLog file for its own executable, and a couple of system files, but did so gracefully -- allowing me to skip the offending files.

Total copy time, start to finish for 15.1 GB was just over 12 minutes. However, whether my D: drive is still bootable now that I overwrote all the files on it with ZTree is unknown. It's past one in the morning now, and I'm running out of patience for experimentation.

The main thing is... I know now that my system is capable of sustained data transfer from one drive to another in excess of 1.25 GB per minute, so Drive Image's copy speed of less than 1/4 that speed while it is actually copying, and 1/10 that speed for the overall process would appear to be a problem in software, not hardware.

My $13 copy of "Ghost" turned out to be a Norton AntiVirus CD, so I still don't have that software for comparison. The lady who sold it to me was very nice, realized she had made an error, and has already put the correct CD in the mail to me, so I'll know later this week if Ghost will do any better for me.

tanstaafl

"There Ain't No Such Thing As A Free Lunch"
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#37015 - 27/08/2001 15:55 Re: OT: Drive Image 4.0 [Re: jimhogan]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5539
Loc: Ajijic, Mexico
Interested to hear what develops.

I was wrong! We did indeed have IIS enabled on our Windows 2000 Advanced Server. And the IIS was indeed infected with Code Red, actually Code Red II.

This caused the file server to overload the processor in the Cisco router. Our chief technical officer for the entire corporation (more than 1,000 radio stations, over 50,000 employees)logged onto our service remotely through a modem connected to the console port of the router and had our CE start disconnecting the router ports one at a time and all of a sudden processor activity went from 100% usage to 5% usage when the cable to the Windows 2000 server was disconnected. After that, we turned off the IIS service, reconnected the cable, and everybody was happy.

Jim, it looked like you (and some others here) called it exactly right. I, and my CE, owe you a big debt of gratitude.

tanstaafl.

"There Ain't No Such Thing As A Free Lunch"
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#37016 - 27/08/2001 16:40 Re: OT: Drive Image 4.0 [Re: tanstaafl.]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31563
Loc: Seattle, WA
Jim, it looked like you (and some others here) called it exactly right. I, and my CE, owe you a big debt of gratitude.

By my calculations, Drakino gets the credit for being the first to suggest Code Red might be the culprit.

I'm responsible for exactly one public-facing web server here, and this discussion made me double-check it. Thankfully, I had patched the server almost immediately after the ISAPI buffer overflow exploit was discovered, so the server was never hit with anything.

Although it's funny because my BlackICE logs show dozens of code red attempts against the server every day.

When I first set up the web server, I did something fairly paranoid. I physically isolated it from our local LAN. We dial out to the internet on a firewalled line that's a totally different line than this server.

In order to update files on the server, we have to use FTP. It doesn't even run the FrontPage extensions (because those have vulnerabilities as well). This is a hassle, because it means that I have to physically walk up to the server to inspect or make any modifications to it.

At first, I thought I might be being too paranoid. Now I'm glad. Even if the server is 100 percent compromised, there is no way for a hacker to use it to compromise any other part of the network.

___________
Tony Fabris
_________________________
Tony Fabris

Top
Page 2 of 2 < 1 2