<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:859009705;
mso-list-template-ids:-53680196;}
@list l0:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:1937398149;
mso-list-template-ids:972035278;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">I was also able to get the xfs_info after finally getting the drive remounted:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D">[root@ncb-sv-016 ~]# xfs_info /home<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D">meta-data=/dev/sdb1 isize=256 agcount=70, agsize=268435455 blks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D"> = sectsz=512 attr=2, projid32bit=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D">data = bsize=4096 blocks=18554637056, imaxpct=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D"> = sunit=0 swidth=0 blks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D">naming =version 2 bsize=4096 ascii-ci=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D">log =internal bsize=4096 blocks=521728, version=2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D"> = sectsz=512 sunit=0 blks, lazy-count=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas;color:#1F497D">realtime =none extsz=4096 blocks=0, rtextents=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Earl, Joshua P <br>
<b>Sent:</b> Tuesday, September 15, 2015 1:53 PM<br>
<b>To:</b> 'xfs@oss.sgi.com' <xfs@oss.sgi.com><br>
<b>Subject:</b> xfsxyncd in 'D' state<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hello, I hope I’m writing to the correct list. I’ve recently run into a problem, which has me stumped. I’m running a cluster which shares an xfs filesystem to 10 nodes via nfs. This has been working for almost two years. However, I’ve
been running into trouble with the drive where if anything tries to write to it at certain times it will simply hang, and every process trying to write will also hang and go into the ‘D’ state. For example (just editing a text file with emacs):<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@ncb-sv-016 ~]# ps aux|grep D<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">root 2216 0.0 0.0 0 0 ? D 11:28 0:00 [xfssyncd/sdb1]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">archana 7708 0.0 0.0 249700 13352 pts/0 D 11:35 0:00 emacs things<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">root 11453 0.0 0.0 103312 868 pts/1 S+ 12:47 0:00 grep D<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal">This will remain like this for hours. Can’t remount/unmount drive (sends the unmount command into ‘D’ state)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have no idea what’s going on or how to fix it, but I’m hoping you guys might be able to point me in the right direction. Here is the info that’s requested in the FAQ:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">kernel version (uname -a)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Linux ncb-sv-016.ducom.edu 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">xfsprogs version (xfs_repair -V)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">xfs_repair version 3.1.1<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">number of CPUs<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">16<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">contents of /proc/meminfo<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">contents of /proc/mounts<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">contents of /proc/partitions<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Attached except for mounts (not currently in the /proc directory, attached the fstab instead (hopefully helpful?)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">RAID layout (hardware and/or software) The RAID-6 is the problem one<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">u0 RAID-1 OK - - - 3725.28 RiW ON
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">u1 RAID-6 OK - - 64K 70780.3 RiW ON
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">u2 SPARE OK - - - 3726.01 - OFF
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">VPort Status Unit Size Type Phy Encl-Slot Model<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p8 OK u0 3.63 TB SATA - /c0/e0/slt0 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p9 OK u1 3.63 TB SATA - /c0/e0/slt4 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p10 OK u1 3.63 TB SATA - /c0/e0/slt8 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p11 OK u1 3.63 TB SATA - /c0/e0/slt12 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p12 OK u1 3.63 TB SATA - /c0/e0/slt16 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p13 OK u1 3.63 TB SATA - /c0/e0/slt20 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p14 OK u0 3.63 TB SATA - /c0/e0/slt1 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p15 OK u1 3.63 TB SATA - /c0/e0/slt5 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p16 OK u1 3.63 TB SATA - /c0/e0/slt9 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p17 OK u1 3.63 TB SATA - /c0/e0/slt13 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p18 OK u1 3.63 TB SATA - /c0/e0/slt17 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p19 OK u1 3.63 TB SATA - /c0/e0/slt21 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p20 OK u1 3.63 TB SATA - /c0/e0/slt2 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p21 OK u1 3.63 TB SATA - /c0/e0/slt6 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p22 OK u1 3.63 TB SATA - /c0/e0/slt10 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p23 OK u1 3.63 TB SATA - /c0/e0/slt14 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p24 OK u1 3.63 TB SATA - /c0/e0/slt18 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p25 OK u1 3.63 TB SATA - /c0/e0/slt22 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p26 OK u1 3.63 TB SATA - /c0/e0/slt3 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p27 OK u1 3.63 TB SATA - /c0/e0/slt7 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p28 OK u1 3.63 TB SATA - /c0/e0/slt11 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p29 OK u1 3.63 TB SATA - /c0/e0/slt15 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p30 OK u1 3.63 TB SATA - /c0/e0/slt19 WDC WD4000FYYZ-01UL
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">p31 OK u2 3.63 TB SATA - /c0/e0/slt23 WDC WD4000FYYZ-01UL<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">LVM configuration<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">I don’t *<b>think</b>* these are in an LVM.. I could be wrong.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">type of disks you are using<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Models included above in raid config.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">write cache status of drives<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">size of BBWC and mode it is running in<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">xfs_info output on the filesystem in question<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">For the above three questions I’m not sure how to get the the cache status of the drives, or what the BBWC is? xfs_info won’t currently run (I’m waiting on the drive to unmount) but I
ran an xfs_check and an xfs_repair –n and no errors were shown.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:19.2pt;text-indent:-.25in;line-height:14.3pt;mso-list:l1 level1 lfo2;background:white">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Wingdings;color:black"><span style="mso-list:Ignore">§<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">dmesg output showing all error messages and stack traces<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:4.8pt;margin-right:0in;margin-bottom:6.0pt;margin-left:0in;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples
of:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:38.4pt;text-indent:-.25in;line-height:14.3pt;mso-list:l0 level1 lfo4;background:white">
<![if !supportLists]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">iostat -x -d -m 5<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">[root@ncb-sv-016 ~]# iostat -x -d -m 5<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Linux 2.6.32-358.23.2.el6.x86_64 (ncb-sv-016.ducom.edu) 09/15/2015 _x86_64_ (16 CPU)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.29 3.61 5.78 3.58 0.10 0.03 28.27 0.05 5.19 2.39 2.24<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 1.02 8.66 31.50 3.91 0.33 0.12 26.14 5.94 167.54 27.47 97.25<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 1.60 0.00 2.00 0.00 0.01 14.40 0.01 4.30 4.30 0.86<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 0.80 0.00 0.03 64.00 6.46 6332.75 1250.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 3.60 0.00 7.00 0.00 0.04 12.11 0.02 0.60 1.77 1.24<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.00 0.00 0.03 64.00 6.28 6256.60 1000.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 0.00 0.00 0.40 0.00 0.00 8.00 0.00 42.50 12.00 0.48<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.20 0.00 0.04 64.00 5.86 5846.33 833.33 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 0.60 0.00 0.60 0.00 0.00 16.00 0.01 12.67 12.67 0.76<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.00 0.00 0.03 64.00 6.86 5725.20 1000.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.00 0.00 0.03 64.00 6.06 5459.00 1000.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 4.00 0.00 1.60 0.00 0.02 26.00 0.01 6.75 6.50 1.04<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.00 0.00 0.03 51.20 7.05 5670.40 1000.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 29.40 0.00 2.60 0.00 0.12 98.46 0.01 4.54 4.08 1.06<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.20 0.00 0.03 53.33 6.54 7428.50 833.33 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 9.60 0.00 15.80 0.00 0.10 12.86 0.57 35.82 3.37 5.32<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.00 0.00 0.03 64.00 6.30 5889.20 1000.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 7.80 0.00 12.80 0.00 0.08 12.38 0.74 58.09 15.06 19.28<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.20 0.00 0.04 64.00 6.49 6140.83 833.33 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 4.80 0.00 3.20 0.00 0.03 20.00 0.01 0.06 3.12 1.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 0.80 0.00 0.03 64.00 5.10 6489.25 1250.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.02 152.00 103.00 2.06<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.00 0.00 0.03 64.00 6.75 5791.00 1000.20 100.02<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 61.20 0.00 11.80 0.00 0.29 49.49 0.01 0.88 0.69 0.82<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.40 0.00 0.04 64.00 6.37 5569.71 714.14 99.98<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 0.40 0.00 0.60 0.00 0.00 13.33 0.01 24.33 24.33 1.46<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.60 0.00 0.05 64.00 5.77 5162.00 625.12 100.02<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 0.00 0.00 0.40 0.00 0.00 8.00 0.00 3.00 1.50 0.06<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 0.80 0.00 0.03 64.00 5.08 3428.50 1250.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 1.60 0.00 0.80 0.00 0.01 24.00 0.01 10.75 10.75 0.86<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.40 0.00 0.04 64.00 5.86 3932.14 714.29 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 2.00 0.00 4.80 0.00 0.03 11.33 0.01 2.21 2.08 1.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.40 0.00 0.04 64.00 5.60 3992.71 714.29 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 5.00 0.00 18.20 0.00 0.09 10.20 0.02 1.13 0.03 0.06<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.40 0.00 0.04 64.00 5.44 4208.86 714.29 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 1.40 0.00 0.60 0.00 0.01 26.67 0.02 27.00 27.00 1.62<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.40 0.00 0.04 64.00 5.22 4325.43 714.29 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 0.60 0.00 0.40 0.00 0.00 20.00 0.01 15.50 15.50 0.62<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.60 0.00 0.05 64.00 5.06 4022.75 625.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 0.80 0.00 0.03 64.00 5.08 3495.50 1250.00 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sda 0.00 1.60 0.00 1.60 0.00 0.01 12.00 0.07 42.88 42.50 6.80<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">sdb 0.00 0.00 0.00 1.40 0.00 0.04 64.00 5.82 3894.71 714.29 100.00<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;margin-left:38.4pt;text-indent:-.25in;line-height:14.3pt;mso-list:l0 level1 lfo4;background:white">
<![if !supportLists]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">vmstat 5<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">[root@ncb-sv-016 ~]# vmstat 5<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">r b swpd free buff cache si so bi bo in cs us sy id wa st<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125768 19456 405396 0 0 28 10 421 150 0 0 99 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125520 19464 405396 0 0 0 34 6679 13281 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">1 0 0 126124896 19472 405392 0 0 0 38 6718 13310 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125312 19472 405400 0 0 0 74 6658 13256 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125440 19480 405392 0 0 0 60 6664 13291 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">2 0 0 126125440 19480 405400 0 0 0 26 6660 13272 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125680 19488 405400 0 0 0 30 6659 13282 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">2 0 0 126125696 19496 405396 0 0 0 117 6686 13298 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">1 0 0 126125568 19496 405400 0 0 0 33 6661 13287 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125816 19504 405400 0 0 0 30 6663 13271 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">1 0 0 126125816 19504 405400 0 0 0 27 6659 13285 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125696 19512 405400 0 0 0 75 6670 13269 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125816 19520 405400 0 0 0 55 6671 13286 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">2 0 0 126125696 19528 405396 0 0 0 34 6670 13284 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125272 19528 405400 0 0 0 26 6700 13298 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">0 0 0 126125408 19536 405400 0 0 0 61 6660 13277 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:1.2pt;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:Consolas;color:black">1 0 0 126125536 19544 405392 0 0 0 98 6677 13281 0 0 100 0 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:4.8pt;margin-right:0in;margin-bottom:6.0pt;margin-left:0in;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">can give us insight into the IO and memory utilisation of your machine at the time of the problem.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:4.8pt;margin-right:0in;margin-bottom:6.0pt;margin-left:0in;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">If the filesystem is hanging, then capture the output of the dmesg command after running:<o:p></o:p></span></p>
<div style="border:dashed #2F6FAB 1.0pt;padding:12.0pt 12.0pt 12.0pt 12.0pt">
<p class="MsoNormal" style="line-height:13.2pt;background:#F9F9F9"><span style="font-size:9.5pt;font-family:"Courier New";color:black"># echo w > /proc/sysrq-trigger<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.2pt;background:#F9F9F9"><span style="font-size:9.5pt;font-family:"Courier New";color:black"># dmesg<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:4.8pt;margin-right:0in;margin-bottom:6.0pt;margin-left:0in;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:4.8pt;margin-right:0in;margin-bottom:6.0pt;margin-left:0in;line-height:14.3pt;background:white">
<span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Attached<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Josh Earl, MS<o:p></o:p></p>
<p class="MsoNormal">Research Instructor<o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Drexel College of Medicine<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span style="color:black">Center for Advanced Microbial Processing (CAMP)<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span style="color:black">Institute of Molecular Medicine and Infectious Disease<o:p></o:p></span></p>
<p class="MsoNormal">(215) 762-8133<span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<hr>
<div>
<p style="font-size:8pt; font-style: italic; line-height:10pt; font-family:Arial, Helvetica, sans-serif">
This email and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this email communication by others is strictly prohibited.
If you are not the intended recipient, please notify the sender immediately and delete all copies. Thank you for your cooperation.
</p>
</div>
</body>
</html>