[EHPweb] quake.usgs.gov not updating

[Robert Simpson]

Hi Tom and Jill,

In case there is a Northern California earthquake today, you should be  
aware that the quake.usgs.gov webpages are not updating. The 3  
Earthquake Hazards Program "quake" servers ehznorth.wr.usgs.gov,  
ehzden.cr.usgs.gov, and ehzeast.er.usgs.gov aren't getting earthquake  
information from the QDDS hubs.

The earthquake.usgs.gov webpages appear to be working OK at the moment.

I reported network problems last night to x4444 and got an email from  
Jeff Brody reporting that there appears to be a problem with one of  
the Domain Name Servers.  It's a mystery to me why the failure of a  
single domain name server would affect our quake servers in Denver and  
Reston, but I suppose it's not impossible.  (If so, we have another  
single point of failure.

I also call the ENS NOSC number last night (866-376-6672) thinking  
that the problem might be a USGS network problem.  The problem was  
assigned a ticket number (see attached).

Sorry, but I don't know what to do next.  If there's an earthquake,  
we're in trouble.

Bob
   simpson at usgs.gov
   (650)-329-4865  (home)
   (650)-322-9322  (office)


On Oct 19, 2008, at 12:39 AM, Jeff A Brody wrote:

>
>
>
>
> Hi Bob,
>
> I received your message this evening at about 11:30pm.  The problem  
> appears
> to be related a problem with the Domain Name Server - 130.118.4.2.
> I'm confirming this.
> For the interim, referencing the 130.118.4.7 server works.
>
> Jeff
>

---------------------------
From: "Tait, Michael R" <michael.r.tait at verizonbusiness.com>
Date: October 19, 2008 12:27:53 AM PDT
To: Robert Simpson <simpson at usgs.gov>
Cc: David Oppenheimer H <oppen at usgs.gov>, Wald Lisa <lisa at usgs.gov>,  
Christopher Bidwell <cbidwell at usgs.gov>, doinoc at lists.mci.com
Subject: RE: Requesting trace routes from Source to Destination

Mr. Simpson,

I’ve opened ticket 2008101901243.   I haven’t determined or  
isolated the cause of the reported problem.  At this time I am  
referring to 2nd level to review ticket in the morning.

Thanks,

Michael Tait
Verizon Business ESN NOSC
866-376-6672



From: Robert Simpson [mailto:simpson at usgs.gov]
Sent: Sunday, October 19, 2008 2:13 AM
To: Tait, Michael R
Cc: David Oppenheimer H; Wald Lisa; Christopher Bidwell
Subject: Re: Requesting trace routes from Source to Destination

Hi Mike,

Here are the 3 hubs:

host qdds1.wr.usgs.gov
qdds1.wr.usgs.gov has address 130.118.43.18

host qdds2.er.usgs.gov
qdds2.er.usgs.gov has address 130.11.56.112

host dmc.iris.washington.edu
dmc.iris.washington.edu has address 128.95.166.129

And here are 3 traceroutes, the last one which is still working...

pascal:9>  traceroute ehznorth.wr.usgs.gov
traceroute to ehznorth.wr.usgs.gov (130.118.61.120), 64 hops max, 40  
byte packets
  1  salt5-l3-vlan22 (130.118.22.215)  4.167 ms  0.411 ms  0.299 ms
  2  ehznorth (130.118.61.120)  0.181 ms  0.242 ms  0.178 ms

pascal:10>  traceroute ehzeast.er.usgs.gov
traceroute to ehzeast.er.usgs.gov (130.11.44.114), 64 hops max, 40  
byte packets
  1  salt5-l3-vlan22 (130.118.22.215)  4.205 ms  0.354 ms  0.241 ms
  2  mpj1-ca-f000.usgs.net (130.118.12.116)  2.939 ms  0.904 ms  0.848  
ms
  3  198.187.220.102 (198.187.220.102)  66.595 ms  66.583 ms  66.405 ms
  4  130.11.7.249 (130.11.7.249)  66.497 ms  66.483 ms  66.395 ms
  5  ehzeast.er.usgs.gov (130.11.44.114)  66.593 ms  66.677 ms  66.585  
ms

pascal:11>  traceroute ehzden.cr.usgs.gov
traceroute to ehzden.cr.usgs.gov (137.227.224.73), 64 hops max, 40  
byte packets
  1  salt5-l3-vlan22 (130.118.22.215)  4.109 ms  0.342 ms  0.241 ms
  2  mp1-ca-f100.usgs.net (130.118.12.117)  3.161 ms  1.009 ms  0.808 ms
  3  198.187.220.123 (198.187.220.123)  28.620 ms  28.677 ms  28.699 ms
  4  * * *
  5  * * *
  6  * * *
  7  * * *
  8  * * *
  9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *


On Oct 18, 2008, at 10:59 PM, Tait, Michael R wrote:


Mr. Simpson

To help isolate any possible capacity, latency or routing issue; can  
you please supply us with trace routes from your source ip address to  
each server?   Could you also supply the ip addresses of the hubs that  
feed the servers?

Thanks,

Michael Tait
Verizon Business ESN NOSC
866-376-6672