<div dir="ltr">Hello Dave, thank you for the response.  I got some recommendations on the ceph-users list that essentially pointed to the problem with vm.swappiness=0 and its new behavior - described here <a href="https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel/">https://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel/</a><div><br></div><div>Basically setting it to 0 creates these OOM conditions due to never swapping anything out.  So I changed these settings right away:</div><div><br></div><div><p style="margin:0px;padding:0px;border:0px;outline:0px;font-size:16px;vertical-align:baseline;color:rgb(0,0,0);font-family:'Segoe UI',helvetica,arial,sans-serif;line-height:21px;background-image:initial;background-repeat:initial">sysctl vm.swappiness=20 (can probably be 1 as per article)</p><p style="margin:0px;padding:0px;border:0px;outline:0px;font-size:16px;vertical-align:baseline;color:rgb(0,0,0);font-family:'Segoe UI',helvetica,arial,sans-serif;line-height:21px;background-image:initial;background-repeat:initial">sysctl vm.min_free_kbytes=262144</p><p style="margin:0px;padding:0px;border:0px;outline:0px;font-size:16px;vertical-align:baseline;color:rgb(0,0,0);font-family:'Segoe UI',helvetica,arial,sans-serif;line-height:21px;background-image:initial;background-repeat:initial"><br></p><p style="margin:0px;padding:0px;border:0px;outline:0px;font-size:16px;vertical-align:baseline;color:rgb(0,0,0);font-family:'Segoe UI',helvetica,arial,sans-serif;line-height:21px;background-image:initial;background-repeat:initial">So far no issues, but I need to wait a week to see if anything shows up.  Thank you for reviewing the error codes.</p><p style="margin:0px;padding:0px;border:0px;outline:0px;font-size:16px;vertical-align:baseline;color:rgb(0,0,0);font-family:'Segoe UI',helvetica,arial,sans-serif;line-height:21px;background-image:initial;background-repeat:initial"><br></p><p style="margin:0px;padding:0px;border:0px;outline:0px;font-size:16px;vertical-align:baseline;color:rgb(0,0,0);font-family:'Segoe UI',helvetica,arial,sans-serif;line-height:21px;background-image:initial;background-repeat:initial">Alex</p></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 3, 2015 at 7:51 PM, Dave Chinner <span dir="ltr"><<a href="mailto:david@fromorbit.com" target="_blank">david@fromorbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Fri, Jul 03, 2015 at 05:07:29AM -0400, Alex Gorbachev wrote:<br>
> Hello, we are seeing this and similar errors on multiple Supermicro nodes<br>
> running Ceph.  OS is Ubuntu 14.04.2 with kernel 4.1<br>
><br>
> Thank you for any info and troubleshooting advice.<br>
<br>
</span>Nothing to suggest that this is an XFS problem. Memory reclaim<br>
triggered by network stack memory pressure is causing inode<br>
eviction. While removing the page cache it's falling over in<br>
the generic truncate code doing a radix tree lookup. That's all<br>
generic code - XFS never touches the page cache radix tree directly.<br>
<br>
I haven't seen this before - is this a new problem since you<br>
upgraded your kernel to 4.1? Is it repeatable? if yes to both, then<br>
a bisect may be in order to isolate the problematic commit...<br>
<br>
Cheers,<br>
<br>
Dave.<br>
<span class="HOEnZb"><font color="#888888">--<br>
Dave Chinner<br>
<a href="mailto:david@fromorbit.com">david@fromorbit.com</a><br>
<br>
_______________________________________________<br>
xfs mailing list<br>
<a href="mailto:xfs@oss.sgi.com">xfs@oss.sgi.com</a><br>
<a href="http://oss.sgi.com/mailman/listinfo/xfs" rel="noreferrer" target="_blank">http://oss.sgi.com/mailman/listinfo/xfs</a><br>
</font></span></blockquote></div><br></div>