<div dir="ltr">Hi Ben, <div><br></div><div style>Thanks again for your help. This should be all the relevant information that you requested:</div><div style><br></div><div style>The workload is classic ETL. The problem shows up when trying to refresh the DWH. </div>
<div style><br></div><div style><div> <b>uname -a</b></div><div>Linux 3.0.42-0.7-default #1 SMP Tue Oct 9 11:58:45 UTC 2012 (a8dc443) x86_64 x86_64 x86_64 GNU/Linux</div><div><br></div><div><b>xfs_repair -V</b></div><div>
xfs_repair version 3.1.8</div><div><br></div><div>Number of CPU: 8</div><div><br></div><div>processor : 7</div><div>vendor_id : GenuineIntel</div><div>cpu family : 6</div><div>model : 23</div><div>
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz</div><div>stepping : 6</div><div>cpu MHz : 2999.800</div><div>cache size : 6144 KB</div><div>physical id : 1</div><div>siblings : 4</div>
<div>core id : 3</div><div>cpu cores : 4</div><div>apicid : 7</div><div>initial apicid : 7</div><div>fpu : yes</div><div>fpu_exception : yes</div><div>cpuid level : 10</div><div>wp : yes</div>
<div>flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dtherm tpr_shadow vnmi flexpriority</div>
<div>bogomips : 6003.48</div><div>clflush size : 64</div><div>cache_alignment : 64</div><div>address sizes : 38 bits physical, 48 bits virtual</div><div>power management:</div><div><br></div><div><b>cat /proc/meminfo</b></div>
<div>MemTotal: 24738004 kB</div><div>MemFree: 903492 kB</div><div>Buffers: 364 kB</div><div>Cached: 22131464 kB</div><div>SwapCached: 28044 kB</div><div>Active: 15011808 kB</div>
<div>Inactive: 7694912 kB</div><div>Active(anon): 12375396 kB</div><div>Inactive(anon): 3024204 kB</div><div>Active(file): 2636412 kB</div><div>Inactive(file): 4670708 kB</div><div>Unevictable: 5160 kB</div>
<div>Mlocked: 5160 kB</div><div>SwapTotal: 18874364 kB</div><div>SwapFree: 13429788 kB</div><div>Dirty: 916 kB</div><div>Writeback: 0 kB</div><div>AnonPages: 559160 kB</div>
<div>Mapped: 13143856 kB</div><div>Shmem: 14821620 kB</div><div>Slab: 618944 kB</div><div>SReclaimable: 499808 kB</div><div>SUnreclaim: 119136 kB</div><div>KernelStack: 5512 kB</div>
<div>PageTables: 64124 kB</div><div>NFS_Unstable: 0 kB</div><div>Bounce: 0 kB</div><div>WritebackTmp: 0 kB</div><div>CommitLimit: 31243364 kB</div><div>Committed_AS: 31705548 kB</div>
<div>VmallocTotal: 34359738367 kB</div><div>VmallocUsed: 336820 kB</div><div>VmallocChunk: 34359375216 kB</div><div>HardwareCorrupted: 0 kB</div><div>AnonHugePages: 258048 kB</div><div>HugePages_Total: 0</div>
<div>HugePages_Free: 0</div><div>HugePages_Rsvd: 0</div><div>HugePages_Surp: 0</div><div>Hugepagesize: 2048 kB</div><div>DirectMap4k: 129392 kB</div><div>DirectMap2M: 25034752 kB</div><div>
<br></div><div><b>cat /proc/mounts</b></div><div>rootfs / rootfs rw 0 0</div><div>udev /dev tmpfs rw,relatime,nr_inodes=0,mode=755 0 0</div><div>tmpfs /dev/shm tmpfs rw,relatime 0 0</div><div>/dev/mapper/system-root / xfs rw,relatime,delaylog,noquota 0 0</div>
<div>proc /proc proc rw,relatime 0 0</div><div>sysfs /sys sysfs rw,relatime 0 0</div><div>devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0</div><div>debugfs /sys/kernel/debug debugfs rw,relatime 0 0</div>
<div>/dev/cciss/c0d0p1 /boot ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0</div><div>/dev/mapper/system-export /export xfs rw,relatime,delaylog,noquota 0 0</div><div>/dev/mapper/system-opt /opt xfs rw,relatime,delaylog,noquota 0 0</div>
<div>/dev/mapper/system-var /var xfs rw,relatime,delaylog,noquota 0 0</div><div>/dev/mapper/system-var /tmp xfs rw,relatime,delaylog,noquota 0 0</div><div style>... etc</div><div><br></div><div><b> cat /proc/partitions</b></div>
<div><br></div><div>major minor #blocks name</div><div><br></div><div> 104 0 143338560 cciss/c0d0</div><div> 104 1 104391 cciss/c0d0p1</div><div> 104 2 143227507 cciss/c0d0p2</div><div> 253 0 10485760 dm-0</div>
<div> 253 1 10485760 dm-1</div><div> 253 2 8388608 dm-2</div><div> 253 3 18874368 dm-3</div><div> 253 4 5242880 dm-4</div><div> 8 48 536870912 sdd</div><div> 8 0 536870912 sda</div>
<div> 8 64 536870912 sde</div><div> 8 32 536870912 sdc</div><div> 8 80 536870912 sdf</div><div> 8 96 536870912 sdg</div><div> 8 16 536870912 sdb</div><div> 8 112 536870912 sdh</div>
<div> 8 128 536870912 sdi</div><div> 8 160 536870912 sdk</div><div> 8 144 536870912 sdj</div><div> 8 176 536870912 sdl</div><div> 8 192 536870912 sdm</div><div> 8 224 536870912 sdo</div>
<div> 8 240 536870912 sdp</div><div> 8 208 536870912 sdn</div><div> 65 0 536870912 sdq</div><div> 65 16 536870912 sdr</div><div> 65 32 536870912 sds</div><div> 65 48 536870912 sdt</div>
<div> 65 64 536870912 sdu</div><div> 65 80 536870912 sdv</div><div> 65 96 536870912 sdw</div><div> 65 112 536870912 sdx</div><div> 65 128 536870912 sdy</div><div> 65 144 536870912 sdz</div>
<div> 65 160 536870912 sdaa</div><div> 65 176 536870912 sdab</div><div> 65 192 536870912 sdac</div><div> 65 208 536870912 sdad</div><div> 65 224 536870912 sdae</div><div> 65 240 536870912 sdaf</div>
<div> 253 5 536870912 dm-5</div><div> 253 6 536870912 dm-6</div><div style> etc....</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Feb 22, 2013 at 9:25 PM, Ben Myers <span dir="ltr"><<a href="mailto:bpm@sgi.com" target="_blank">bpm@sgi.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Erik,<br>
<div><div class="h5"><br>
On Fri, Feb 22, 2013 at 02:04:17PM +0100, Erik Knight wrote:<br>
> We've recently noticed that our system is experiencing extreme performance<br>
> problems when running large workloads. The problem seems to come from<br>
> excessive System CPU time. Specifically dozens of xfsaild threads. We used<br>
> to have SSD drives but recently switched to HDD, so some of us are thinking<br>
> that there may be a configuration issue within XFS that is optimized for<br>
> SSD but performs terribly slow on HDD.<br>
><br>
> Can anyone explain what these threads do, what would cause so many of them<br>
> to be running simultaneously or consume so much CPU?<br>
<br>
</div></div>AIL stands for Active Item List. These guys sync metadata which has been<br>
logged to its final location on disk. You'll have one daemon per filesystem.<br>
<br>
If you have a very metadata intensive workload they could get a workout. It<br>
would help to know a bit more about your workload and configuration. Can you<br>
provide the relevant information listed here?<br>
<br>
<a href="http://www.xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F" target="_blank">http://www.xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F</a><br>
<br>
Regards,<br>
Ben<br>
</blockquote></div><br></div>