[EHPweb] EIDS problems

[Jeremy M Fee]

Yes, but there were filesystem errors.  The fsck on the logical volume  
group appears to have fixed the problem.  Strange that it would happen  
on multiple systems at the same time.

eids2 and eids3 are both working properly and communicating with other  
hubs.


Jeremy


On Oct 5, 2009, at 11:20 AM, Christopher J Bidwell wrote:

> If you type mount, does it show the filesystem as read-write?
> --------------
> Thanks,
>
> Chris Bidwell, RHCT
> Web Admin
> Geologic Hazards Team
> 303-273-8642
> cbidwell at usgs.gov
> (Sent via Blackberry)
>
>
> ----- Original Message -----
> From: "David Oppenheimer" [oppen at usgs.gov]
> Sent: 10/05/2009 10:14 AM MST
> To: Jeremy Fee; Stanley Silverman; "'Stan Schwarz'" <stan7826 at gmail.com 
> >
> Cc: Christopher Bidwell; "'EHPweb'" <ehpweb at geohazards.usgs.gov>
> Subject: RE: [EHPweb] EIDS problems
>
>
>
> We can log in root. Want us to do a fsck?
>
> -------------------------------------------------------
> David Oppenheimer                   office:650.329.4792
> U.S. Geological Survey              fax:   650.329.4732
> 345 Middlefield Road.-MS 977    email: oppen at usgs.gov
> Menlo Park, CA 94025
>
>
> -----Original Message-----
> From: Jeremy M Fee [mailto:jmfee at usgs.gov]
> Sent: Monday, October 05, 2009 10:05 AM
> To: David Oppenheimer; Silverman Stan; Stan Schwarz
> Cc: Chris Bidwell; EHPweb
> Subject: Re: [EHPweb] EIDS problems
>
> eids2 and eids3 are having strange filesystem errors (saying read only
> although it isn't).  It sounds like the filesystems may have been
> corrupted and need to be repaired (according to google results).
>
> Stan Silverman and Stan Schwarz need to check out /var/log/messages
> (as root) on eids2.wr.usgs.gov and eids3.gps.caltech.edu to find out
> what is going on.
>
> Jeremy
>
>
> On Oct 5, 2009, at 10:54 AM, David Oppenheimer wrote:
>
>> EIDS2 is spewing the following message to the console.
>>
>> EXT3 - fs error (device dm0) in start_transaction. Journal aborted.
>>
>> I can log into the machine as user ppicker from another machine, and
>> it is
>> responsive, but I am not sure I can log in via the console.
>>
>> Let me know if you need me to do something.
>>
>> -David
>>
>> -------------------------------------------------------
>> David Oppenheimer                   office:650.329.4792
>> U.S. Geological Survey              fax:   650.329.4732
>> 345 Middlefield Road.-MS 977    email: oppen at usgs.gov
>> Menlo Park, CA 94025
>>
>>
>> -----Original Message-----
>> From: ehpweb-bounces at geohazards.usgs.gov
>> [mailto:ehpweb-bounces at geohazards.usgs.gov] On Behalf Of Jeremy M Fee
>> Sent: Monday, October 05, 2009 7:48 AM
>> To: Chris Bidwell
>> Cc: EHPweb
>> Subject: Re: [EHPweb] EIDS problems
>>
>> Correction, it appears eids1 was not affected by this (whew) and
>> messages have not been interrupted.
>>
>> eids2 and eids3 are still effectively down.
>>
>>
>> Jeremy
>>
>>
>> On Oct 5, 2009, at 7:49 AM, Jeremy M Fee wrote:
>>
>>> It looks like eids1, eids2, and eids3 are having problems running
>>> EIDS and QDDS.  This appears to be down since yesterday.  I can't
>>> access /var/log/messages, which likely contains more information.
>>>
>>>
>>> [eids at eids2 QDDS]$ java -jar QDDS.jar
>>> Creating new log file: 200910051342
>>> Problem in LogIt Constructor: java.io.FileNotFoundException: log/
>>> 200910051342 (Read-only file system)
>>>
>>> [eids at eids2 EIDS]$ java -jar bin/ISTINotifServer.jar
>>> Exception opening log file "log/NotificationServer_20091005.log
>>> (Read-only file system)":  java.io.FileNotFoundException: log/
>>> NotificationServer_20091005.log (Read-only file system)
>>>
>>> [eids at eids2 EIDS]$ touch test.text
>>> touch: cannot touch `test.text': Read-only file system
>>>
>>>
>>> Mount shows / as mounted read-write, and, from what I've read, this
>>> is due to filesystem corruption?
>>> [eids at eids2 EIDS]$ mount
>>> /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
>>> none on /proc type proc (rw)
>>> none on /sys type sysfs (rw)
>>> none on /dev/pts type devpts (rw,gid=5,mode=620)
>>> usbfs on /proc/bus/usb type usbfs (rw)
>>> /dev/sda1 on /boot type ext3 (rw)
>>> none on /dev/shm type tmpfs (rw)
>>> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
>>>
>>>
>>>
>>> Jeremy Fee <jmfee at usgs.gov>
>>> Geologic Hazards Team
>>>
>>
>> _______________________________________________
>> EHPweb mailing list
>> EHPweb at geohazards.usgs.gov
>> https://geohazards.usgs.gov/mailman/listinfo/ehpweb
>>
>