No subject
Tom Christensen
tom.christensen at storagecraft.com
Fri Jun 12 20:02:51 CDT 2015
We've run into a bit of an issue with xfs running Ceph. The following bug details what we are seeing:
https://bugs.launchpad.net/ubuntu/+source/xfs/+bug/1464308
Basically the Ceph OSD process gets hung in dstate due to the traceback in the bug.
Here is additional info gathered:
xfs bmap output for a random directory
https://gist.github.com/dmmatson/e864252c7ff346df954a
attr -l of the file dchinner indicated from the xfs bmap output
(attr -l)
(6:11:41 PM) dmatson: Attribute "cephos.spill_out" has a 2 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:11:45 PM) dmatson: Attribute "ceph.snapset at 3" has a 263 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:11:49 PM) dmatson: Attribute "ceph.snapset at 2" has a 1131 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:11:53 PM) dmatson: Attribute "ceph.snapset at 1" has a 2048 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:11:56 PM) dmatson: Attribute "ceph._" has a 259 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
(6:12:00 PM) dmatson: Attribute "ceph.snapset" has a 2048 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5
xfs_bmap -vp of same file
rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5:
(6:13:21 PM) dmatson: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
(6:13:25 PM) dmatson: 0: [0..8191]: 2944471376..2944479567 16 (24445776..24453967) 8192 00000
uname -a
Linux str-slc-02-01 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64
x86_64 GNU/Linux
xfsprogs version
xfs_repair version 3.1.9
number of cpus
2x6 core
/proc/meminfo
MemTotal: 264123244 kB
MemFree: 115763776 kB
MemAvailable: 118825516 kB
Buffers: 8624 kB
Cached: 5331460 kB
SwapCached: 0 kB
Active: 137378292 kB
Inactive: 4797536 kB
Active(anon): 136839564 kB
Inactive(anon): 5160 kB
Active(file): 538728 kB
Inactive(file): 4792376 kB
Unevictable: 8576 kB
Mlocked: 8896 kB
SwapTotal: 34058232 kB
SwapFree: 34058232 kB
Dirty: 116400 kB
Writeback: 3360 kB
AnonPages: 136847296 kB
Mapped: 3003196 kB
Shmem: 2776 kB
Slab: 1348116 kB
SReclaimable: 581188 kB
SUnreclaim: 766928 kB
KernelStack: 1788400 kB
PageTables: 505196 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 166119852 kB
Committed_AS: 259735308 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 908732 kB
VmallocChunk: 34221991936 kB
HardwareCorrupted: 0 kB
AnonHugePages: 34844672 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 137500 kB
DirectMap2M: 5070848 kB
DirectMap1G: 265289728 kB
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=132050356k,nr_inodes=33012589,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=26412328k,mode=755 0 0
/dev/mapper/system-root / ext4 rw,relatime,discard,errors=remount-ro,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
/dev/sda1 /boot ext3 rw,relatime,data=ordered 0 0
systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0
/dev/mapper/e0364357-e2e0-4b5e-92c3-abbea655ca77 /var/lib/ceph/osd/ceph-293 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/a7745ae8-1b83-4cb5-892b-4cc8d27ed5b2 /var/lib/ceph/osd/ceph-302 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/60cecd8c-42fd-4cbf-9264-ead7319846d3 /var/lib/ceph/osd/ceph-295 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/c3ab6273-a646-40fe-bbf3-c4559bf4202a /var/lib/ceph/osd/ceph-298 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/3dfceed9-6563-46c9-b174-7173f73874ae /var/lib/ceph/osd/ceph-301 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/5765a366-3908-41ef-b2ab-dd10db6e8c29 /var/lib/ceph/osd/ceph-299 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/bb5b11a9-cc4d-49b8-a909-c8e84a88f0ee /var/lib/ceph/osd/ceph-316 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/4d3165f9-d879-426f-b7ef-6a959a09a1bf /var/lib/ceph/osd/ceph-313 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/2559e439-9540-40eb-99a6-cebbe2663cd9 /var/lib/ceph/tmp/mnt.BeT4xM xfs rw,relatime,attr2,inode64,noquota 0 0
/dev/mapper/b492dfe2-103c-44b6-83ad-8dc35a9818c0 /var/lib/ceph/osd/ceph-303 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/e5b359e2-bb98-40a5-a285-612b8fb2b2a6 /var/lib/ceph/osd/ceph-314 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/6b45df8f-b6c0-4ed7-b791-a2694b3212ec /var/lib/ceph/osd/ceph-312 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/abcd86ff-57d0-4ef0-8ee7-1a13408b3b2c /var/lib/ceph/osd/ceph-322 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/a75ed2c0-cc39-4063-bb1e-202d47798499 /var/lib/ceph/osd/ceph-317 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/4d788f02-2720-4a50-8bac-e5a4cdf99441 /var/lib/ceph/osd/ceph-321 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/8d179b94-a071-4dab-927b-b9d2ed98451d /var/lib/ceph/osd/ceph-308 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/b781ebc5-7259-4cf2-8fd5-c608c61fe0e2 /var/lib/ceph/osd/ceph-318 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/deaa099e-9791-4c03-826e-6386eb2c4fff /var/lib/ceph/osd/ceph-323 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/bcbf31e3-7f17-49aa-9259-6dfe8e0f04bc /var/lib/ceph/osd/ceph-320 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/d2b9331f-bcfa-4c98-a0b7-5ab1ec93ee02 /var/lib/ceph/osd/ceph-311 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/0c4172f9-9acb-4c88-bc4e-1b6eb898ea28 /var/lib/ceph/osd/ceph-325 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/252ee856-0376-44ae-b6d7-03f930f8d082 /var/lib/ceph/osd/ceph-328 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/d4bcc76c-20c3-4253-bd05-472f3eb7df96 /var/lib/ceph/osd/ceph-319 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/2ca3f1d3-cf59-49df-a008-0bb6c549587b /var/lib/ceph/osd/ceph-315 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/16df1fdc-819b-4593-ac4f-80f3fcac6054 /var/lib/ceph/osd/ceph-306 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/ce83d72e-029b-40b6-9d59-3b0ebadea2c6 /var/lib/ceph/osd/ceph-324 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/86e078fe-6fa7-466c-83eb-14c53b531320 /var/lib/ceph/osd/ceph-307 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/368ef2f0-da43-48ee-bd91-2bfba95aea5a /var/lib/ceph/osd/ceph-310 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/ac67fd93-8145-40ce-96bb-812cec326306 /var/lib/ceph/osd/ceph-327 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/098e2832-e9d4-4b84-9fa9-acfa407a3afc /var/lib/ceph/osd/ceph-309 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/f1588472-430d-4db4-83dc-31e1ec1d2ba5 /var/lib/ceph/osd/ceph-326 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/fb64b259-7d70-4859-824e-b6ee7683495d /var/lib/ceph/osd/ceph-305 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/2559e439-9540-40eb-99a6-cebbe2663cd9 /var/lib/ceph/osd/ceph-300 xfs rw,noatime,attr2,inode64,noquota 0 0
/dev/mapper/ae8b44b1-25ac-4aa9-b987-9c57c3410ec6 /var/lib/ceph/osd/ceph-304 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/c198f472-4806-4c9c-aa79-b8e1b5554cdf /var/lib/ceph/osd/ceph-296 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/65d17dba-4c30-4e95-a0b5-577002712b6a /var/lib/ceph/osd/ceph-297 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
/dev/mapper/1f3dc9e8-0d84-4951-8c9b-48b6a6be83b8 /var/lib/ceph/osd/ceph-294 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0
major minor #blocks name
8 0 125034840 sda
8 1 291840 sda1
8 2 1 sda2
8 5 124740608 sda5
252 0 124235776 dm-0
252 1 503808 dm-1
8 32 2930266584 sdc
8 33 2920025543 sdc1
8 34 10238976 sdc2
8 128 2930266584 sdi
8 129 2920025543 sdi1
8 130 10238976 sdi2
8 96 2930266584 sdg
8 97 2920025543 sdg1
8 98 10238976 sdg2
8 112 2930266584 sdh
8 113 2920025543 sdh1
8 114 10238976 sdh2
8 176 2930266584 sdl
8 177 2920025543 sdl1
8 178 10238976 sdl2
8 144 2930266584 sdj
8 145 2920025543 sdj1
8 146 10238976 sdj2
8 240 2930266584 sdp
8 241 2920025543 sdp1
8 242 10238976 sdp2
8 224 2930266584 sdo
8 225 2920025543 sdo1
8 226 10238976 sdo2
8 80 2930266584 sdf
8 81 2920025543 sdf1
8 82 10238976 sdf2
8 48 2930266584 sdd
8 49 2920025543 sdd1
8 50 10238976 sdd2
65 48 2930266584 sdt
65 49 2920025543 sdt1
65 50 10238976 sdt2
8 64 2930266584 sde
8 65 2920025543 sde1
8 66 10238976 sde2
65 0 2930266584 sdq
65 1 2920025543 sdq1
65 2 10238976 sdq2
8 192 2930266584 sdm
8 193 2920025543 sdm1
8 194 10238976 sdm2
65 32 2930266584 sds
65 33 2920025543 sds1
65 34 10238976 sds2
65 64 2930266584 sdu
65 65 2920025543 sdu1
65 66 10238976 sdu2
65 16 2930266584 sdr
65 17 2920025543 sdr1
65 18 10238976 sdr2
65 160 2930266584 sdaa
65 161 2920025543 sdaa1
65 162 10238976 sdaa2
8 160 2930266584 sdk
8 161 2920025543 sdk1
8 162 10238976 sdk2
65 240 2930266584 sdaf
65 241 2920025543 sdaf1
65 242 10238976 sdaf2
8 16 2930266584 sdb
8 17 2920025543 sdb1
8 18 10238976 sdb2
65 176 2930266584 sdab
65 177 2920025543 sdab1
65 178 10238976 sdab2
65 96 2930266584 sdw
65 97 2920025543 sdw1
65 98 10238976 sdw2
65 128 2930266584 sdy
65 129 2920025543 sdy1
65 130 10238976 sdy2
66 16 2930266584 sdah
66 17 2920025543 sdah1
66 18 10238976 sdah2
8 208 2930266584 sdn
8 209 2920025543 sdn1
8 210 10238976 sdn2
65 144 2930266584 sdz
65 145 2920025543 sdz1
65 146 10238976 sdz2
65 80 2930266584 sdv
65 81 2920025543 sdv1
65 82 10238976 sdv2
65 192 2930266584 sdac
65 193 2920025543 sdac1
65 194 10238976 sdac2
65 224 2930266584 sdae
65 225 2920025543 sdae1
65 226 10238976 sdae2
65 112 2930266584 sdx
65 113 2920025543 sdx1
65 114 10238976 sdx2
66 0 2930266584 sdag
66 1 2920025543 sdag1
66 2 10238976 sdag2
65 208 2930266584 sdad
65 209 2920025543 sdad1
65 210 10238976 sdad2
66 64 2930266584 sdak
66 65 2920025543 sdak1
66 66 10238976 sdak2
66 48 2930266584 sdaj
66 49 2920025543 sdaj1
66 50 10238976 sdaj2
66 32 2930266584 sdai
66 33 2920025543 sdai1
66 34 10238976 sdai2
252 2 10238976 dm-2
252 3 2920025543 dm-3
252 4 10238976 dm-4
252 5 2920025543 dm-5
252 6 2920025543 dm-6
252 7 10238976 dm-7
252 8 2920025543 dm-8
252 9 10238976 dm-9
252 10 2920025543 dm-10
252 11 10238976 dm-11
252 12 10238976 dm-12
252 13 2920025543 dm-13
252 14 10238976 dm-14
252 15 2920025543 dm-15
252 16 10238976 dm-16
252 17 2920025543 dm-17
252 18 2920025543 dm-18
252 19 10238976 dm-19
252 20 2920025543 dm-20
252 21 10238976 dm-21
252 22 10238976 dm-22
252 23 2920025543 dm-23
252 24 10238976 dm-24
252 25 2920025543 dm-25
252 26 2920025543 dm-26
252 27 10238976 dm-27
252 28 10238976 dm-28
252 29 2920025543 dm-29
252 31 2920025543 dm-31
252 30 10238976 dm-30
252 32 10238976 dm-32
252 33 2920025543 dm-33
252 34 10238976 dm-34
252 35 2920025543 dm-35
252 36 2920025543 dm-36
252 37 10238976 dm-37
252 38 2920025543 dm-38
252 39 10238976 dm-39
252 40 2920025543 dm-40
252 41 2920025543 dm-41
252 42 2920025543 dm-42
252 43 10238976 dm-43
252 44 10238976 dm-44
252 45 10238976 dm-45
252 46 10238976 dm-46
252 47 2920025543 dm-47
252 48 2920025543 dm-48
252 49 10238976 dm-49
252 50 2920025543 dm-50
252 51 10238976 dm-51
252 52 2920025543 dm-52
252 53 10238976 dm-53
252 54 10238976 dm-54
252 55 2920025543 dm-55
252 56 2920025543 dm-56
252 57 10238976 dm-57
252 59 2920025543 dm-59
252 58 10238976 dm-58
252 60 10238976 dm-60
252 61 2920025543 dm-61
252 62 2920025543 dm-62
252 63 10238976 dm-63
252 64 2920025543 dm-64
252 65 10238976 dm-65
252 66 2920025543 dm-66
252 67 10238976 dm-67
252 68 2920025543 dm-68
252 69 10238976 dm-69
252 70 10238976 dm-70
252 71 2920025543 dm-71
252 72 10238976 dm-72
252 73 2920025543 dm-73
RAID layout: None
LVM configuration: None
HGST 3TB
meta-data=/dev/mapper/63e3300b-6b7b-41a0-bdf2-b64ec3c20c51 isize=2048 agcount=32, agsize=22812700 blks
= sectsz=4096 attr=2
data = bsize=4096 blocks=730006385, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=356448, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Some additional info can be found in the following ubuntu bug:
________________________________
Tom Christensen | Senior Software Engineer | StorageCraft Technology Corporation<http://www.storagecraft.com>
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2774 | Mobile: 385.230.2422
________________________________
If you are not the intended recipient of this message, be advised that any dissemination or copying of this message is prohibited.
If you received this message erroneously, please notify the sender and delete it, together with any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20150613/c9d9b843/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dmesg.out
Type: application/octet-stream
Size: 207835 bytes
Desc: dmesg.out
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20150613/c9d9b843/attachment-0001.obj>
More information about the xfs
mailing list