xfs
[Top] [All Lists]

Re: XFS filesystem on EC2 instance corrupts and shuts down

To: Shrinath M <shrinath.m@xxxxxxxxxx>
Subject: Re: XFS filesystem on EC2 instance corrupts and shuts down
From: Ben Myers <bpm@xxxxxxx>
Date: Wed, 13 Mar 2013 13:24:35 -0500
Cc: Ric Wheeler <rwheeler@xxxxxxxxxx>, Sabyasachi Ruj <sabyasachi.ruj@xxxxxxxxxx>, xfs@xxxxxxxxxxx, Supratik Goswami <supratik.goswami@xxxxxxxxxx>, Vivek Goel <vivek.goel@xxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CAOdS1hngSuHn_HiremLyUS7Qd9eZ68=8arfBuHnEpwXQaBw9Wg@xxxxxxxxxxxxxx>
References: <CAOdS1h=7X4O1O7X8YOwxtLm7G=fc+J+6hJxJ1RKbDmfTZXTpeg@xxxxxxxxxxxxxx> <51373DB8.2020707@xxxxxxxxxx> <CAOdS1hnXGj9puaHxeToqmpK40A-3WvJnM7=5HckpyyZYqZTvEQ@xxxxxxxxxxxxxx> <51373FC1.6010101@xxxxxxxxxx> <CAOurMUeasru6ekDYcvVR1QnaWVJFV+-coZsUG5SgG6LnENBvXg@xxxxxxxxxxxxxx> <513751F2.2060109@xxxxxxxxxx> <CAOdS1hngSuHn_HiremLyUS7Qd9eZ68=8arfBuHnEpwXQaBw9Wg@xxxxxxxxxxxxxx>
User-agent: Mutt/1.5.20 (2009-06-14)
Hey Shrinath,

On Wed, Mar 13, 2013 at 11:37:52PM +0530, Shrinath M wrote:
> Sorry to be asking in dev thread, but Amazon seems to be clueless in this
> case :(
> Can someone tell me where can we find the logs/output of xfs repair after
> this runs?

xfs_repair doesn't keep a separate log file.  All the output is on the command
line.  You'll need to either redirect the output of stdout and stderr to a
file, or keep a screen or console log.

>       We just reboot the machine when we see this and the
> /var/log/messages or dmesg seems to know nothing about what it repaired.

The contents of /var/log/messages could help you to understand why xfs might
have forced shutdown, but won't help you with xfs_repair.

Since you're getting IO errors, it sounds like you have a problem lower in the
stack than the filesystem.  At the filesystem level there isn't much we can do
with a block device that is giving IO errors so we just shut down.  Consider
copying the remote block device to a local one ('dd' might be a good choice for
this) and see if you can get a clean copy.  Then it's time to see about the
filesystem.

Regards,
Ben

> On Wed, Mar 6, 2013 at 7:55 PM, Ric Wheeler <rwheeler@xxxxxxxxxx> wrote:
> 
> > I would suggest contacting Amazon's customer support channel (or the
> > vendor you paid for the Linux instance you are running).
> >
> > XFS developer list is probably not the correct forum to help you debug
> > this :)
> >
> > Good luck!
> >
> > Ric
> >
> >
> >
> > On 03/06/2013 08:12 AM, Supratik Goswami wrote:
> >
> >> Have we created a ticket with AWS ?
> >>
> >> It could be an EBS issue who knows, we need to confirm that first.
> >>
> >> --
> >> Warm Regards
> >>
> >> Supratik
> >>
> >>
> >> On Wed, Mar 6, 2013 at 6:38 PM, Ric Wheeler <rwheeler@xxxxxxxxxx <mailto:
> >> rwheeler@xxxxxxxxxx>> wrote:
> >>
> >>     On 03/06/2013 08:03 AM, Shrinath M wrote:
> >>
> >>
> >>         On Wed, Mar 6, 2013 at 6:29 PM, Ric Wheeler <rwheeler@xxxxxxxxxx
> >>         <mailto:rwheeler@xxxxxxxxxx> <mailto:rwheeler@xxxxxxxxxx
> >>
> >>         <mailto:rwheeler@xxxxxxxxxx>>> wrote:
> >>
> >>             I think that you would need to verify that the Amazon storage
> >> is not
> >>             throwing errors - do your logs show IO errors or issues
> >> before XFS
> >>         hits an
> >>             issue?
> >>
> >>
> >>         No IO errors in /var/log/messages.
> >>         Where else should I be looking?
> >>
> >>
> >>
> >>     Feb 12 19:47:18 ip-100-0-100-1 kernel: [2541168.023638] XFS (md0): I/O
> >>     Error Detected. Shutting down filesystem
> >>
> >>     Is an IO error from MD.
> >>
> >>     I would suggest trying to reproduce without MD in the picture first -
> >>     always best to try to reproduce with the simplest setup first and work
> >>     your way up the complexity ladder,
> >>
> >>     Ric
> >>
> >>
> >>
> >>
> >> ______________________________**_________________
> >> xfs mailing list
> >> xfs@xxxxxxxxxxx
> >> http://oss.sgi.com/mailman/**listinfo/xfs<http://oss.sgi.com/mailman/listinfo/xfs>
> >>
> >
> >
> 
> 
> -- 
> Regards
> *Shrinath.M*

> _______________________________________________
> xfs mailing list
> xfs@xxxxxxxxxxx
> http://oss.sgi.com/mailman/listinfo/xfs

<Prev in Thread] Current Thread [Next in Thread>