<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;"><span lang="en-US"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt;"><font color="#1F497D">We've run into a bit of an issue with xfs running Ceph. The following
bug details what we are seeing: </font></span></font></span><br>
<span lang="en-US"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt;"><font color="#1F497D"><a href="https://bugs.launchpad.net/ubuntu/+source/xfs/+bug/1464308" target="_blank">https://bugs.launchpad.net/ubuntu/+source/xfs/+bug/1464308</a><br>
<br>
Basically the Ceph OSD process gets hung in dstate due to the traceback in the bug.<br>
<br>
Here is additional info gathered:<br>
<br>
xfs bmap output for a random directory<br>
https://gist.github.com/dmmatson/e864252c7ff346df954a</font></span></font></span><br>
<br>
attr -l of the file dchinner indicated from the xfs bmap output<br>
<span lang="en-US"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt;">(attr -l)<br>
<font color="#204A87" size="2"><span style="font-size:10pt;">(6:11:41 PM) </span>
</font><font color="#204A87"><b>dmatson: </b></font>Attribute "cephos.spill_out" has a 2 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5<br>
<font color="#204A87" size="2"><span style="font-size:10pt;">(6:11:45 PM) </span>
</font><font color="#204A87"><b>dmatson: </b></font>Attribute "ceph.snapset@3" has a 263 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5<br>
<font color="#204A87" size="2"><span style="font-size:10pt;">(6:11:49 PM) </span>
</font><font color="#204A87"><b>dmatson: </b></font>Attribute "ceph.snapset@2" has a 1131 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5<br>
<font color="#204A87" size="2"><span style="font-size:10pt;">(6:11:53 PM) </span>
</font><font color="#204A87"><b>dmatson: </b></font>Attribute "ceph.snapset@1" has a 2048 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5<br>
<font color="#204A87" size="2"><span style="font-size:10pt;">(6:11:56 PM) </span>
</font><font color="#204A87"><b>dmatson: </b></font>Attribute "ceph._" has a 259 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5<br>
<font color="#204A87" size="2"><span style="font-size:10pt;">(6:12:00 PM) </span>
</font><font color="#204A87"><b>dmatson: </b></font>Attribute "ceph.snapset" has a 2048 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5<br>
<br>
xfs_bmap -vp of same file<br>
</span></font></span><br>
<span lang="en-US"><font face="Calibri,sans-serif" size="2"><span style="font-size:11pt;">rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5:<br>
<font color="#204A87" size="2"><span style="font-size:10pt;">(6:13:21 PM) </span>
</font><font color="#204A87"><b>dmatson: </b></font>EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS<br>
<font color="#204A87" size="2"><span style="font-size:10pt;">(6:13:25 PM) </span>
</font><font color="#204A87"><b>dmatson: </b></font>0: [0..8191]: 2944471376..2944479567 16 (24445776..24453967) 8192 00000<br>
<br>
</span></font></span>uname -a<br>
Linux str-slc-02-01 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64
<br>
x86_64 GNU/Linux<br>
<br>
xfsprogs version<br>
xfs_repair version 3.1.9<br>
<br>
number of cpus<br>
2x6 core<br>
<br>
/proc/meminfo<br>
MemTotal: 264123244 kB<br>
MemFree: 115763776 kB<br>
MemAvailable: 118825516 kB<br>
Buffers: 8624 kB<br>
Cached: 5331460 kB<br>
SwapCached: 0 kB<br>
Active: 137378292 kB<br>
Inactive: 4797536 kB<br>
Active(anon): 136839564 kB<br>
Inactive(anon): 5160 kB<br>
Active(file): 538728 kB<br>
Inactive(file): 4792376 kB<br>
Unevictable: 8576 kB<br>
Mlocked: 8896 kB<br>
SwapTotal: 34058232 kB<br>
SwapFree: 34058232 kB<br>
Dirty: 116400 kB<br>
Writeback: 3360 kB<br>
AnonPages: 136847296 kB<br>
Mapped: 3003196 kB<br>
Shmem: 2776 kB<br>
Slab: 1348116 kB<br>
SReclaimable: 581188 kB<br>
SUnreclaim: 766928 kB<br>
KernelStack: 1788400 kB<br>
PageTables: 505196 kB<br>
NFS_Unstable: 0 kB<br>
Bounce: 0 kB<br>
WritebackTmp: 0 kB<br>
CommitLimit: 166119852 kB<br>
Committed_AS: 259735308 kB<br>
VmallocTotal: 34359738367 kB<br>
VmallocUsed: 908732 kB<br>
VmallocChunk: 34221991936 kB<br>
HardwareCorrupted: 0 kB<br>
AnonHugePages: 34844672 kB<br>
HugePages_Total: 0<br>
HugePages_Free: 0<br>
HugePages_Rsvd: 0<br>
HugePages_Surp: 0<br>
Hugepagesize: 2048 kB<br>
DirectMap4k: 137500 kB<br>
DirectMap2M: 5070848 kB<br>
DirectMap1G: 265289728 kB<br>
<br>
rootfs / rootfs rw 0 0<br>
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0<br>
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0<br>
udev /dev devtmpfs rw,relatime,size=132050356k,nr_inodes=33012589,mode=755 0 0<br>
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0<br>
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=26412328k,mode=755 0 0<br>
/dev/mapper/system-root / ext4 rw,relatime,discard,errors=remount-ro,data=ordered 0 0<br>
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0<br>
none /sys/fs/fuse/connections fusectl rw,relatime 0 0<br>
none /sys/kernel/debug debugfs rw,relatime 0 0<br>
none /sys/kernel/security securityfs rw,relatime 0 0<br>
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0<br>
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0<br>
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0<br>
none /sys/fs/pstore pstore rw,relatime 0 0<br>
/dev/sda1 /boot ext3 rw,relatime,data=ordered 0 0<br>
systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0<br>
/dev/mapper/e0364357-e2e0-4b5e-92c3-abbea655ca77 /var/lib/ceph/osd/ceph-293 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/a7745ae8-1b83-4cb5-892b-4cc8d27ed5b2 /var/lib/ceph/osd/ceph-302 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/60cecd8c-42fd-4cbf-9264-ead7319846d3 /var/lib/ceph/osd/ceph-295 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/c3ab6273-a646-40fe-bbf3-c4559bf4202a /var/lib/ceph/osd/ceph-298 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/3dfceed9-6563-46c9-b174-7173f73874ae /var/lib/ceph/osd/ceph-301 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/5765a366-3908-41ef-b2ab-dd10db6e8c29 /var/lib/ceph/osd/ceph-299 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/bb5b11a9-cc4d-49b8-a909-c8e84a88f0ee /var/lib/ceph/osd/ceph-316 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/4d3165f9-d879-426f-b7ef-6a959a09a1bf /var/lib/ceph/osd/ceph-313 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/2559e439-9540-40eb-99a6-cebbe2663cd9 /var/lib/ceph/tmp/mnt.BeT4xM xfs rw,relatime,attr2,inode64,noquota 0 0<br>
/dev/mapper/b492dfe2-103c-44b6-83ad-8dc35a9818c0 /var/lib/ceph/osd/ceph-303 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/e5b359e2-bb98-40a5-a285-612b8fb2b2a6 /var/lib/ceph/osd/ceph-314 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/6b45df8f-b6c0-4ed7-b791-a2694b3212ec /var/lib/ceph/osd/ceph-312 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/abcd86ff-57d0-4ef0-8ee7-1a13408b3b2c /var/lib/ceph/osd/ceph-322 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/a75ed2c0-cc39-4063-bb1e-202d47798499 /var/lib/ceph/osd/ceph-317 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/4d788f02-2720-4a50-8bac-e5a4cdf99441 /var/lib/ceph/osd/ceph-321 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/8d179b94-a071-4dab-927b-b9d2ed98451d /var/lib/ceph/osd/ceph-308 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/b781ebc5-7259-4cf2-8fd5-c608c61fe0e2 /var/lib/ceph/osd/ceph-318 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/deaa099e-9791-4c03-826e-6386eb2c4fff /var/lib/ceph/osd/ceph-323 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/bcbf31e3-7f17-49aa-9259-6dfe8e0f04bc /var/lib/ceph/osd/ceph-320 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/d2b9331f-bcfa-4c98-a0b7-5ab1ec93ee02 /var/lib/ceph/osd/ceph-311 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/0c4172f9-9acb-4c88-bc4e-1b6eb898ea28 /var/lib/ceph/osd/ceph-325 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/252ee856-0376-44ae-b6d7-03f930f8d082 /var/lib/ceph/osd/ceph-328 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/d4bcc76c-20c3-4253-bd05-472f3eb7df96 /var/lib/ceph/osd/ceph-319 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/2ca3f1d3-cf59-49df-a008-0bb6c549587b /var/lib/ceph/osd/ceph-315 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/16df1fdc-819b-4593-ac4f-80f3fcac6054 /var/lib/ceph/osd/ceph-306 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/ce83d72e-029b-40b6-9d59-3b0ebadea2c6 /var/lib/ceph/osd/ceph-324 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/86e078fe-6fa7-466c-83eb-14c53b531320 /var/lib/ceph/osd/ceph-307 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/368ef2f0-da43-48ee-bd91-2bfba95aea5a /var/lib/ceph/osd/ceph-310 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/ac67fd93-8145-40ce-96bb-812cec326306 /var/lib/ceph/osd/ceph-327 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/098e2832-e9d4-4b84-9fa9-acfa407a3afc /var/lib/ceph/osd/ceph-309 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/f1588472-430d-4db4-83dc-31e1ec1d2ba5 /var/lib/ceph/osd/ceph-326 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/fb64b259-7d70-4859-824e-b6ee7683495d /var/lib/ceph/osd/ceph-305 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/2559e439-9540-40eb-99a6-cebbe2663cd9 /var/lib/ceph/osd/ceph-300 xfs rw,noatime,attr2,inode64,noquota 0 0<br>
/dev/mapper/ae8b44b1-25ac-4aa9-b987-9c57c3410ec6 /var/lib/ceph/osd/ceph-304 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/c198f472-4806-4c9c-aa79-b8e1b5554cdf /var/lib/ceph/osd/ceph-296 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/65d17dba-4c30-4e95-a0b5-577002712b6a /var/lib/ceph/osd/ceph-297 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
/dev/mapper/1f3dc9e8-0d84-4951-8c9b-48b6a6be83b8 /var/lib/ceph/osd/ceph-294 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0<br>
<br>
major minor #blocks name<br>
<br>
8 0 125034840 sda<br>
8 1 291840 sda1<br>
8 2 1 sda2<br>
8 5 124740608 sda5<br>
252 0 124235776 dm-0<br>
252 1 503808 dm-1<br>
8 32 2930266584 sdc<br>
8 33 2920025543 sdc1<br>
8 34 10238976 sdc2<br>
8 128 2930266584 sdi<br>
8 129 2920025543 sdi1<br>
8 130 10238976 sdi2<br>
8 96 2930266584 sdg<br>
8 97 2920025543 sdg1<br>
8 98 10238976 sdg2<br>
8 112 2930266584 sdh<br>
8 113 2920025543 sdh1<br>
8 114 10238976 sdh2<br>
8 176 2930266584 sdl<br>
8 177 2920025543 sdl1<br>
8 178 10238976 sdl2<br>
8 144 2930266584 sdj<br>
8 145 2920025543 sdj1<br>
8 146 10238976 sdj2<br>
8 240 2930266584 sdp<br>
8 241 2920025543 sdp1<br>
8 242 10238976 sdp2<br>
8 224 2930266584 sdo<br>
8 225 2920025543 sdo1<br>
8 226 10238976 sdo2<br>
8 80 2930266584 sdf<br>
8 81 2920025543 sdf1<br>
8 82 10238976 sdf2<br>
8 48 2930266584 sdd<br>
8 49 2920025543 sdd1<br>
8 50 10238976 sdd2<br>
65 48 2930266584 sdt<br>
65 49 2920025543 sdt1<br>
65 50 10238976 sdt2<br>
8 64 2930266584 sde<br>
8 65 2920025543 sde1<br>
8 66 10238976 sde2<br>
65 0 2930266584 sdq<br>
65 1 2920025543 sdq1<br>
65 2 10238976 sdq2<br>
8 192 2930266584 sdm<br>
8 193 2920025543 sdm1<br>
8 194 10238976 sdm2<br>
65 32 2930266584 sds<br>
65 33 2920025543 sds1<br>
65 34 10238976 sds2<br>
65 64 2930266584 sdu<br>
65 65 2920025543 sdu1<br>
65 66 10238976 sdu2<br>
65 16 2930266584 sdr<br>
65 17 2920025543 sdr1<br>
65 18 10238976 sdr2<br>
65 160 2930266584 sdaa<br>
65 161 2920025543 sdaa1<br>
65 162 10238976 sdaa2<br>
8 160 2930266584 sdk<br>
8 161 2920025543 sdk1<br>
8 162 10238976 sdk2<br>
65 240 2930266584 sdaf<br>
65 241 2920025543 sdaf1<br>
65 242 10238976 sdaf2<br>
8 16 2930266584 sdb<br>
8 17 2920025543 sdb1<br>
8 18 10238976 sdb2<br>
65 176 2930266584 sdab<br>
65 177 2920025543 sdab1<br>
65 178 10238976 sdab2<br>
65 96 2930266584 sdw<br>
65 97 2920025543 sdw1<br>
65 98 10238976 sdw2<br>
65 128 2930266584 sdy<br>
65 129 2920025543 sdy1<br>
65 130 10238976 sdy2<br>
66 16 2930266584 sdah<br>
66 17 2920025543 sdah1<br>
66 18 10238976 sdah2<br>
8 208 2930266584 sdn<br>
8 209 2920025543 sdn1<br>
8 210 10238976 sdn2<br>
65 144 2930266584 sdz<br>
65 145 2920025543 sdz1<br>
65 146 10238976 sdz2<br>
65 80 2930266584 sdv<br>
65 81 2920025543 sdv1<br>
65 82 10238976 sdv2<br>
65 192 2930266584 sdac<br>
65 193 2920025543 sdac1<br>
65 194 10238976 sdac2<br>
65 224 2930266584 sdae<br>
65 225 2920025543 sdae1<br>
65 226 10238976 sdae2<br>
65 112 2930266584 sdx<br>
65 113 2920025543 sdx1<br>
65 114 10238976 sdx2<br>
66 0 2930266584 sdag<br>
66 1 2920025543 sdag1<br>
66 2 10238976 sdag2<br>
65 208 2930266584 sdad<br>
65 209 2920025543 sdad1<br>
65 210 10238976 sdad2<br>
66 64 2930266584 sdak<br>
66 65 2920025543 sdak1<br>
66 66 10238976 sdak2<br>
66 48 2930266584 sdaj<br>
66 49 2920025543 sdaj1<br>
66 50 10238976 sdaj2<br>
66 32 2930266584 sdai<br>
66 33 2920025543 sdai1<br>
66 34 10238976 sdai2<br>
252 2 10238976 dm-2<br>
252 3 2920025543 dm-3<br>
252 4 10238976 dm-4<br>
252 5 2920025543 dm-5<br>
252 6 2920025543 dm-6<br>
252 7 10238976 dm-7<br>
252 8 2920025543 dm-8<br>
252 9 10238976 dm-9<br>
252 10 2920025543 dm-10<br>
252 11 10238976 dm-11<br>
252 12 10238976 dm-12<br>
252 13 2920025543 dm-13<br>
252 14 10238976 dm-14<br>
252 15 2920025543 dm-15<br>
252 16 10238976 dm-16<br>
252 17 2920025543 dm-17<br>
252 18 2920025543 dm-18<br>
252 19 10238976 dm-19<br>
252 20 2920025543 dm-20<br>
252 21 10238976 dm-21<br>
252 22 10238976 dm-22<br>
252 23 2920025543 dm-23<br>
252 24 10238976 dm-24<br>
252 25 2920025543 dm-25<br>
252 26 2920025543 dm-26<br>
252 27 10238976 dm-27<br>
252 28 10238976 dm-28<br>
252 29 2920025543 dm-29<br>
252 31 2920025543 dm-31<br>
252 30 10238976 dm-30<br>
252 32 10238976 dm-32<br>
252 33 2920025543 dm-33<br>
252 34 10238976 dm-34<br>
252 35 2920025543 dm-35<br>
252 36 2920025543 dm-36<br>
252 37 10238976 dm-37<br>
252 38 2920025543 dm-38<br>
252 39 10238976 dm-39<br>
252 40 2920025543 dm-40<br>
252 41 2920025543 dm-41<br>
252 42 2920025543 dm-42<br>
252 43 10238976 dm-43<br>
252 44 10238976 dm-44<br>
252 45 10238976 dm-45<br>
252 46 10238976 dm-46<br>
252 47 2920025543 dm-47<br>
252 48 2920025543 dm-48<br>
252 49 10238976 dm-49<br>
252 50 2920025543 dm-50<br>
252 51 10238976 dm-51<br>
252 52 2920025543 dm-52<br>
252 53 10238976 dm-53<br>
252 54 10238976 dm-54<br>
252 55 2920025543 dm-55<br>
252 56 2920025543 dm-56<br>
252 57 10238976 dm-57<br>
252 59 2920025543 dm-59<br>
252 58 10238976 dm-58<br>
252 60 10238976 dm-60<br>
252 61 2920025543 dm-61<br>
252 62 2920025543 dm-62<br>
252 63 10238976 dm-63<br>
252 64 2920025543 dm-64<br>
252 65 10238976 dm-65<br>
252 66 2920025543 dm-66<br>
252 67 10238976 dm-67<br>
252 68 2920025543 dm-68<br>
252 69 10238976 dm-69<br>
252 70 10238976 dm-70<br>
252 71 2920025543 dm-71<br>
252 72 10238976 dm-72<br>
252 73 2920025543 dm-73<br>
<br>
RAID layout: None<br>
<br>
LVM configuration: None<br>
<br>
HGST 3TB<br>
<br>
meta-data=/dev/mapper/63e3300b-6b7b-41a0-bdf2-b64ec3c20c51 isize=2048 agcount=32, agsize=22812700 blks<br>
= sectsz=4096 attr=2<br>
data = bsize=4096 blocks=730006385, imaxpct=5<br>
= sunit=0 swidth=0 blks<br>
naming =version 2 bsize=4096 ascii-ci=0<br>
log =internal bsize=4096 blocks=356448, version=2<br>
= sectsz=4096 sunit=1 blks, lazy-count=1<br>
realtime =none extsz=4096 blocks=0, rtextents=0<br>
<br>
Some additional info can be found in the following ubuntu bug:<br>
<br>
<br>
</div>
<table border="0" cellspacing="0" cellpadding="0" width="auto">
<tbody>
<tr>
<td valign="top" width="auto"><font face="Calibri, Arial, Helvetica, sans-serif" size="2" color="#063565">
<hr>
<strong>Tom Christensen </strong>| Senior Software Engineer | <a href="http://www.storagecraft.com" vlink="#063565">
<font face="Calibri, Arial, Helvetica, sans-serif" size="2" color="#063565"><strong>StorageCraft Technology Corporation</strong></a>
<br>
380 Data Drive Suite 300 | Draper | Utah | 84020 <br>
<strong>Office: </strong>801.871.2774 | <strong>Mobile: </strong>385.230.2422 <br>
</font>
<hr>
<div></div>
</font></td>
</tr>
</tbody>
</table>
<div><font face="Calibri, Arial, Helvetica, sans-serif" size="1.5" color="#666666">If you are not the intended recipient of this message, be advised that any dissemination or copying of this message is prohibited.
<br>
If you received this message erroneously, please notify the sender and delete it, together with any attachments.
<br>
</font></div>
</body>
</html>