Horrid Site Response

[Foxbat]

I have been trying to get into SatGuys for the last 20 minutes. I'm seeing 28% packet loss between here and SatelliteGuys.US. The traceroute looks good from SBN until between Dallas and SatGuys.US:

Tracing route to www.satelliteguys.us [65.99.220.89]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  adsl-70-224-77-79.dsl.sbndin.ameritech.net [70.224.77.79]
  2     8 ms     7 ms     7 ms  adsl-70-224-79-254.dsl.sbndin.ameritech.net [70.224.79.254]
  3     8 ms     8 ms     7 ms  dist1-vlan50.sbndin.sbcglobal.net [65.43.5.226]
  4     8 ms     7 ms    10 ms  bb1-g8-0.sbndin.sbcglobal.net [65.43.5.115]
  5    11 ms    10 ms    10 ms  bb2-p12-2.chcgil.sbcglobal.net [151.164.43.151]
  6    11 ms    11 ms    10 ms  ex1-p0-0.eqchil.sbcglobal.net [151.164.42.147]
  7    11 ms    11 ms    11 ms  ge-3-2-1.edge2.Chicago1.Level3.net [4.68.111.13]
  8    19 ms    17 ms    22 ms  ae-32-54.ebr2.Chicago1.Level3.net [4.68.101.126]
  9    12 ms    50 ms    53 ms  ae-3.ebr2.Denver1.Level3.net [4.69.132.61]
 10    52 ms    53 ms    54 ms  ae-3.ebr2.Denver1.Level3.net [4.69.132.61]
 11    44 ms     *       54 ms  ae-1-100.ebr1.Denver1.Level3.net [4.69.132.37]
 12    45 ms    56 ms    56 ms  ae-2.ebr1.Dallas1.Level3.net [4.69.132.106]
 13    46 ms    47 ms    46 ms  ae-11-51.car1.Dallas1.Level3.net [4.68.122.13]
 14    45 ms    44 ms    46 ms  unknown.Level3.net [209.246.152.246]
 15    45 ms     *       46 ms  border4.tge3-1-bbnnet1.ext1.dal.pnap.net [216.52.191.30]
 16   164 ms     *        *     206.123.64.38
 17   166 ms   163 ms   164 ms  206.123.64.38
 18   165 ms   165 ms   165 ms  65.99.220.89

Trace complete.

Anyone else having issues?

 

[Scott Greczkowski]

I am as well and have submitted a ticket, its not our server its the connection between pnap.net and our isp.

Tracing route to www.satelliteguys.us [65.99.220.89]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 10 ms 6 ms 11 ms 10.4.136.1
3 6 ms 12 ms 6 ms rkhlsysc01-gex0102000.ct.ri.cox.net [68.9.8.165]
4 13 ms 12 ms 38 ms ip68-9-7-1.ri.ri.cox.net [68.9.7.1]
5 11 ms 11 ms 12 ms provdsrj02-ge500.rd.ri.cox.net [68.9.14.105]
6 10 ms 11 ms 11 ms provbbrj01-ge120.rd.ri.cox.net [68.1.0.58]
7 35 ms 35 ms 35 ms chgobbrj01-so100.r2.ch.cox.net [68.1.0.40]
8 31 ms 31 ms 39 ms 68.1.0.222
9 52 ms 59 ms 51 ms mtc3bbrj02-so112.rd.ok.cox.net [68.1.0.116]
10 57 ms 51 ms 50 ms mtc3bbrj01-ae0.rd.ok.cox.net [68.1.0.120]
11 57 ms 59 ms 56 ms 68.1.0.151
12 58 ms 76 ms 56 ms core101.coxpeer-220.ext1a.dal.pnap.net [216.52.189.165]
13 57 ms 60 ms 54 ms border4.tge3-1-bbnnet1.ext1.dal.pnap.net [216.52.191.30]
14 170 ms 172 ms 173 ms colo4dallas-3.border4.ext1.dal.pnap.net [216.52.189.10]
15 177 ms 175 ms 176 ms 206.123.64.38
16 173 ms 226 ms 177 ms 206.123.64.110
17 173 ms 170 ms 171 ms 65.99.220.89

Trace complete.

 

[Scott Greczkowski]

Just after I submitted this message I got an update from the ISP that they have kicked the rouge router.

Looking better here already.

 

[Foxbat]

I was going to say! I went from 20-50% packet loss and 178ms response to 0% loss and 45-50ms response.

Thanks, Scott!

 

[Scott Greczkowski]

Just FYI I just got another note from the ISP in regards to this issue.

We are aware of the issue and are in conference with our carriers to isolate the issue. We will update you to verify that your serverice has been restored.

 

[Foxbat]

It's interesting that my connection is down from 18 to 14 hops now in addition to being faster. Gotta love that Internet!

 

[charper1]

Glad it wasn't just me.

 

[Foxbat]

Freakin' A, man... Another router problem? I lost SatelliteGuys between 11:00 and 11:40 tonight. Scott needs to get a free month of hosting services for this!

 

[rockymtnhigh]

Freakin' A, man... Another router problem?

Seriously! completely down for about an hour as far as I can tell!

 

[Neutron]

It took me forever to connect this evening.

I couldn't even ping it.

 

[Scott Greczkowski]

This was a server issue. We are still working on it.

Hang tight!

 

[Scott Greczkowski]

We are currently running in safe mode. Only 1 CPU in operation.

 

[Scott Greczkowski]

Now back on all 4 CPU's however running a file system check which is slowing things down.

I may close the site for a little bit why the system processes

 

[damaged]

Saw some extended (10-15 mins) routing issue again tonite with inaps pnap, perhaps you could ask your provider (and maybe to ask their provider) to stop announcing your routes/ip space to inaps pnaps, then you would not have to worry about inap suckage, all routes to you would be handled by the rest of the peers, excluding inap.

 

[Scott Greczkowski]

They are aware of those issues and are working on fixing them. An ETA however is not available.

 

[damaged]

They are aware of those issues and are working on fixing them. An ETA however is not available.

Fair enough, I don't mean to be a pain, just trying to help.

 

[Scott Greczkowski]

No problem didn't think you were being a pain.

All feedback and advice welcome and encouraged.

 

[damaged]

No problem didn't think you were being a pain.

All feedback and advice welcome and encouraged.

 

[Foxbat]

I'm glad damaged knows what he's talking about, 'cause he's way past my wiki... Glad to see everything seems to be back to normal. I hope this isn't a hardware issue, Scott.

 

[damaged]

Now back on all 4 CPU's however running a file system check which is slowing things down.

I may close the site for a little bit why the system processes

If I may, when you run things like that, you can use the nice command, for example, if the job/command is named update-dbase, and you know it is cpu intensive you can run it at a lower priority than the webserver (which is probably set to 0 priority), so to run it and give the web server priority, instead of ruuning './update-dbase', it would become: 'nice -n19 ./update-dbase', this way those jobs won't bog down site response, also good in cronjobs.

do a: 'man nice', but, in case you don't have the manpage, I pasted the relevant part of it below, also you can use renice on an already running process, hope this helps.

If you have the smp tools installed with the programs/modules to adjust affinity, you can even assign those background jobs to a seperate CPU.

man nice output:

Run COMMAND with an adjusted scheduling priority. With no COMMAND, print the current scheduling priority. ADJUST is 10 by default. Range goes from -20 (highest priority) to 19 (lowest).

-ADJUST
increment priority by ADJUST first

-n, --adjustment=ADJUST
same as -ADJUST

BTW, you can also use the command renice to reschedule an already running process.

man renice:
NAME
renice - alter priority of running processes

SYNOPSIS
renice priority [[-p] pid ...] [[-g] pgrp ...] [[-u] user ...]