xfs
[Top] [All Lists]

[no subject]

To: "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Subject:
From: Tom Christensen <tom.christensen@xxxxxxxxxxxxxxxx>
Date: Sat, 13 Jun 2015 01:02:51 +0000
Accept-language: en-US
Cc: "swakefie@xxxxxxxxxx" <swakefie@xxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
Thread-index: AdClcY+xSZ7A/UPhSWeu8RONs4wKFA==
We've run into a bit of an issue with xfs running Ceph.  The following bug details what we are seeing:
https://bugs.launchpad.net/ubuntu/+source/xfs/+bug/1464308

Basically the Ceph OSD process gets hung in dstate due to the traceback in the bug.

Here is additional info gathered:

xfs bmap output for a random directory
https://gist.github.com/dmmatson/e864252c7ff346df954a


attr -l of the file dchinner indicated from the xfs bmap output
(attr -l)
(6:11:41 PM) dmatson: Attribute "cephos.spill_out" has a 2 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:11:45 PM) dmatson: Attribute "ceph.snapset@3" has a 263 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:11:49 PM) dmatson: Attribute "ceph.snapset@2" has a 1131 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:11:53 PM) dmatson: Attribute "ceph.snapset@1" has a 2048 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:11:56 PM) dmatson: Attribute "ceph._" has a 259 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:12:00 PM) dmatson: Attribute "ceph.snapset" has a 2048 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5

xfs_bmap -vp of same file

rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5:
(6:13:21 PM) dmatson: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
(6:13:25 PM) dmatson: 0: [0..8191]: 2944471376..2944479567 16 (24445776..24453967) 8192 00000

uname -a
Linux str-slc-02-01 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64
x86_64 GNU/Linux

xfsprogs version
xfs_repair version 3.1.9

number of cpus
2x6 core

/proc/meminfo
MemTotal:       264123244 kB
MemFree:        115763776 kB
MemAvailable:   118825516 kB
Buffers:            8624 kB
Cached:          5331460 kB
SwapCached:            0 kB
Active:         137378292 kB
Inactive:        4797536 kB
Active(anon):   136839564 kB
Inactive(anon):     5160 kB
Active(file):     538728 kB
Inactive(file):  4792376 kB
Unevictable:        8576 kB
Mlocked:            8896 kB
SwapTotal:      34058232 kB
SwapFree:       34058232 kB
Dirty:            116400 kB
Writeback:          3360 kB
AnonPages:      136847296 kB
Mapped:          3003196 kB
Shmem:              2776 kB
Slab:            1348116 kB
SReclaimable:     581188 kB
SUnreclaim:       766928 kB
KernelStack:     1788400 kB
PageTables:       505196 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    166119852 kB
Committed_AS:   259735308 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      908732 kB
VmallocChunk:   34221991936 kB
HardwareCorrupted:     0 kB
AnonHugePages:  34844672 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      137500 kB
DirectMap2M:     5070848 kB
DirectMap1G:    265289728 kB

rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=132050356k,nr_inodes=33012589,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=26412328k,mode=755 0 0
/dev/mapper/system-root / ext4 rw,relatime,discard,errors=remount-ro,data="" 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
/dev/sda1 /boot ext3 rw,relatime,data="" 0 0
systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0
/dev/mapper/e0364357-e2e0-4b5e-92c3-abbea655ca77 /var/lib/ceph/osd/ceph-293 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/a7745ae8-1b83-4cb5-892b-4cc8d27ed5b2 /var/lib/ceph/osd/ceph-302 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/60cecd8c-42fd-4cbf-9264-ead7319846d3 /var/lib/ceph/osd/ceph-295 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/c3ab6273-a646-40fe-bbf3-c4559bf4202a /var/lib/ceph/osd/ceph-298 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/3dfceed9-6563-46c9-b174-7173f73874ae /var/lib/ceph/osd/ceph-301 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/5765a366-3908-41ef-b2ab-dd10db6e8c29 /var/lib/ceph/osd/ceph-299 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/bb5b11a9-cc4d-49b8-a909-c8e84a88f0ee /var/lib/ceph/osd/ceph-316 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/4d3165f9-d879-426f-b7ef-6a959a09a1bf /var/lib/ceph/osd/ceph-313 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/2559e439-9540-40eb-99a6-cebbe2663cd9 /var/lib/ceph/tmp/mnt.BeT4xM xfs rw,relatime,attr2,inode64,noquota 0 0
/dev/mapper/b492dfe2-103c-44b6-83ad-8dc35a9818c0 /var/lib/ceph/osd/ceph-303 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/e5b359e2-bb98-40a5-a285-612b8fb2b2a6 /var/lib/ceph/osd/ceph-314 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/6b45df8f-b6c0-4ed7-b791-a2694b3212ec /var/lib/ceph/osd/ceph-312 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/abcd86ff-57d0-4ef0-8ee7-1a13408b3b2c /var/lib/ceph/osd/ceph-322 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/a75ed2c0-cc39-4063-bb1e-202d47798499 /var/lib/ceph/osd/ceph-317 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/4d788f02-2720-4a50-8bac-e5a4cdf99441 /var/lib/ceph/osd/ceph-321 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/8d179b94-a071-4dab-927b-b9d2ed98451d /var/lib/ceph/osd/ceph-308 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/b781ebc5-7259-4cf2-8fd5-c608c61fe0e2 /var/lib/ceph/osd/ceph-318 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/deaa099e-9791-4c03-826e-6386eb2c4fff /var/lib/ceph/osd/ceph-323 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/bcbf31e3-7f17-49aa-9259-6dfe8e0f04bc /var/lib/ceph/osd/ceph-320 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/d2b9331f-bcfa-4c98-a0b7-5ab1ec93ee02 /var/lib/ceph/osd/ceph-311 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/0c4172f9-9acb-4c88-bc4e-1b6eb898ea28 /var/lib/ceph/osd/ceph-325 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/252ee856-0376-44ae-b6d7-03f930f8d082 /var/lib/ceph/osd/ceph-328 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/d4bcc76c-20c3-4253-bd05-472f3eb7df96 /var/lib/ceph/osd/ceph-319 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/2ca3f1d3-cf59-49df-a008-0bb6c549587b /var/lib/ceph/osd/ceph-315 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/16df1fdc-819b-4593-ac4f-80f3fcac6054 /var/lib/ceph/osd/ceph-306 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/ce83d72e-029b-40b6-9d59-3b0ebadea2c6 /var/lib/ceph/osd/ceph-324 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/86e078fe-6fa7-466c-83eb-14c53b531320 /var/lib/ceph/osd/ceph-307 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/368ef2f0-da43-48ee-bd91-2bfba95aea5a /var/lib/ceph/osd/ceph-310 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/ac67fd93-8145-40ce-96bb-812cec326306 /var/lib/ceph/osd/ceph-327 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/098e2832-e9d4-4b84-9fa9-acfa407a3afc /var/lib/ceph/osd/ceph-309 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/f1588472-430d-4db4-83dc-31e1ec1d2ba5 /var/lib/ceph/osd/ceph-326 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/fb64b259-7d70-4859-824e-b6ee7683495d /var/lib/ceph/osd/ceph-305 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/2559e439-9540-40eb-99a6-cebbe2663cd9 /var/lib/ceph/osd/ceph-300 xfs rw,noatime,attr2,inode64,noquota 0 0
/dev/mapper/ae8b44b1-25ac-4aa9-b987-9c57c3410ec6 /var/lib/ceph/osd/ceph-304 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/c198f472-4806-4c9c-aa79-b8e1b5554cdf /var/lib/ceph/osd/ceph-296 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/65d17dba-4c30-4e95-a0b5-577002712b6a /var/lib/ceph/osd/ceph-297 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/1f3dc9e8-0d84-4951-8c9b-48b6a6be83b8 /var/lib/ceph/osd/ceph-294 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0

major minor  #blocks  name

   8        0  125034840 sda
   8        1     291840 sda1
   8        2          1 sda2
   8        5  124740608 sda5
 252        0  124235776 dm-0
 252        1     503808 dm-1
   8       32 2930266584 sdc
   8       33 2920025543 sdc1
   8       34   10238976 sdc2
   8      128 2930266584 sdi
   8      129 2920025543 sdi1
   8      130   10238976 sdi2
   8       96 2930266584 sdg
   8       97 2920025543 sdg1
   8       98   10238976 sdg2
   8      112 2930266584 sdh
   8      113 2920025543 sdh1
   8      114   10238976 sdh2
   8      176 2930266584 sdl
   8      177 2920025543 sdl1
   8      178   10238976 sdl2
   8      144 2930266584 sdj
   8      145 2920025543 sdj1
   8      146   10238976 sdj2
   8      240 2930266584 sdp
   8      241 2920025543 sdp1
   8      242   10238976 sdp2
   8      224 2930266584 sdo
   8      225 2920025543 sdo1
   8      226   10238976 sdo2
   8       80 2930266584 sdf
   8       81 2920025543 sdf1
   8       82   10238976 sdf2
   8       48 2930266584 sdd
   8       49 2920025543 sdd1
   8       50   10238976 sdd2
  65       48 2930266584 sdt
  65       49 2920025543 sdt1
  65       50   10238976 sdt2
   8       64 2930266584 sde
   8       65 2920025543 sde1
   8       66   10238976 sde2
  65        0 2930266584 sdq
  65        1 2920025543 sdq1
  65        2   10238976 sdq2
   8      192 2930266584 sdm
   8      193 2920025543 sdm1
   8      194   10238976 sdm2
  65       32 2930266584 sds
  65       33 2920025543 sds1
  65       34   10238976 sds2
  65       64 2930266584 sdu
  65       65 2920025543 sdu1
  65       66   10238976 sdu2
  65       16 2930266584 sdr
  65       17 2920025543 sdr1
  65       18   10238976 sdr2
  65      160 2930266584 sdaa
  65      161 2920025543 sdaa1
  65      162   10238976 sdaa2
   8      160 2930266584 sdk
   8      161 2920025543 sdk1
   8      162   10238976 sdk2
  65      240 2930266584 sdaf
  65      241 2920025543 sdaf1
  65      242   10238976 sdaf2
   8       16 2930266584 sdb
   8       17 2920025543 sdb1
   8       18   10238976 sdb2
  65      176 2930266584 sdab
  65      177 2920025543 sdab1
  65      178   10238976 sdab2
  65       96 2930266584 sdw
  65       97 2920025543 sdw1
  65       98   10238976 sdw2
  65      128 2930266584 sdy
  65      129 2920025543 sdy1
  65      130   10238976 sdy2
  66       16 2930266584 sdah
  66       17 2920025543 sdah1
  66       18   10238976 sdah2
   8      208 2930266584 sdn
   8      209 2920025543 sdn1
   8      210   10238976 sdn2
  65      144 2930266584 sdz
  65      145 2920025543 sdz1
  65      146   10238976 sdz2
  65       80 2930266584 sdv
  65       81 2920025543 sdv1
  65       82   10238976 sdv2
  65      192 2930266584 sdac
  65      193 2920025543 sdac1
  65      194   10238976 sdac2
  65      224 2930266584 sdae
  65      225 2920025543 sdae1
  65      226   10238976 sdae2
  65      112 2930266584 sdx
  65      113 2920025543 sdx1
  65      114   10238976 sdx2
  66        0 2930266584 sdag
  66        1 2920025543 sdag1
  66        2   10238976 sdag2
  65      208 2930266584 sdad
  65      209 2920025543 sdad1
  65      210   10238976 sdad2
  66       64 2930266584 sdak
  66       65 2920025543 sdak1
  66       66   10238976 sdak2
  66       48 2930266584 sdaj
  66       49 2920025543 sdaj1
  66       50   10238976 sdaj2
  66       32 2930266584 sdai
  66       33 2920025543 sdai1
  66       34   10238976 sdai2
 252        2   10238976 dm-2
 252        3 2920025543 dm-3
 252        4   10238976 dm-4
 252        5 2920025543 dm-5
 252        6 2920025543 dm-6
 252        7   10238976 dm-7
 252        8 2920025543 dm-8
 252        9   10238976 dm-9
 252       10 2920025543 dm-10
 252       11   10238976 dm-11
 252       12   10238976 dm-12
 252       13 2920025543 dm-13
 252       14   10238976 dm-14
 252       15 2920025543 dm-15
 252       16   10238976 dm-16
 252       17 2920025543 dm-17
 252       18 2920025543 dm-18
 252       19   10238976 dm-19
 252       20 2920025543 dm-20
 252       21   10238976 dm-21
 252       22   10238976 dm-22
 252       23 2920025543 dm-23
 252       24   10238976 dm-24
 252       25 2920025543 dm-25
 252       26 2920025543 dm-26
 252       27   10238976 dm-27
 252       28   10238976 dm-28
 252       29 2920025543 dm-29
 252       31 2920025543 dm-31
 252       30   10238976 dm-30
 252       32   10238976 dm-32
 252       33 2920025543 dm-33
 252       34   10238976 dm-34
 252       35 2920025543 dm-35
 252       36 2920025543 dm-36
 252       37   10238976 dm-37
 252       38 2920025543 dm-38
 252       39   10238976 dm-39
 252       40 2920025543 dm-40
 252       41 2920025543 dm-41
 252       42 2920025543 dm-42
 252       43   10238976 dm-43
 252       44   10238976 dm-44
 252       45   10238976 dm-45
 252       46   10238976 dm-46
 252       47 2920025543 dm-47
 252       48 2920025543 dm-48
 252       49   10238976 dm-49
 252       50 2920025543 dm-50
 252       51   10238976 dm-51
 252       52 2920025543 dm-52
 252       53   10238976 dm-53
 252       54   10238976 dm-54
 252       55 2920025543 dm-55
 252       56 2920025543 dm-56
 252       57   10238976 dm-57
 252       59 2920025543 dm-59
 252       58   10238976 dm-58
 252       60   10238976 dm-60
 252       61 2920025543 dm-61
 252       62 2920025543 dm-62
 252       63   10238976 dm-63
 252       64 2920025543 dm-64
 252       65   10238976 dm-65
 252       66 2920025543 dm-66
 252       67   10238976 dm-67
 252       68 2920025543 dm-68
 252       69   10238976 dm-69
 252       70   10238976 dm-70
 252       71 2920025543 dm-71
 252       72   10238976 dm-72
 252       73 2920025543 dm-73

RAID layout: None

LVM configuration: None

HGST 3TB

meta-data="" isize=2048   agcount=32, agsize=22812700 blks
         =                       sectsz=4096  attr=2
data     =                       bsize=4096   blocks=730006385, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=356448, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Some additional info can be found in the following ubuntu bug:



Tom Christensen | Senior Software Engineer | StorageCraft Technology Corporation
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2774 | Mobile: 385.230.2422

If you are not the intended recipient of this message, be advised that any dissemination or copying of this message is prohibited.
If you received this message erroneously, please notify the sender and delete it, together with any attachments.

Attachment: dmesg.out
Description: dmesg.out

<Prev in Thread] Current Thread [Next in Thread>