xfs
[Top] [All Lists]

Re: weird quota issue

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: weird quota issue
From: "Weber, Charles (NIH/NIA/IRP) [C]" <weberc@xxxxxxxxxxxxxxx>
Date: Mon, 22 Dec 2014 17:13:06 -0500
Cc: "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <20141222204857.GK24183@dastard>
References: <3BD1EE39-8C26-4C5B-94B5-492422EECEDA@xxxxxxxxxxxx> <20141222204857.GK24183@dastard>
Thanks for replying. The project part is a red herring and I have abandoned it. The only reason project quotas even came up was the winbind/quota issue. UID is fine.
The more interesting part is the way the /proc/self/mounts and mtab/fstab are not coherent.

2 filesystems have identical (cut and paste) setting in fstab. Below results are after setting forcefsck and rebooting.

mount <enter>
/dev/mapper/irphome_vg-home_lv on /home type xfs (rw,delaylog,inode64,nobarrier,logbsize=256k,uquota,prjquota)
/dev/mapper/irphome_vg-imap_lv on /mail type xfs (rw,delaylog,inode64,nobarrier,logbsize=256k,uquota,prjquota)

cat /proc/self/mounts
/dev/mapper/irphome_vg-home_lv /home xfs rw,relatime,attr2,delaylog,nobarrier,inode64,logbsize=256k,sunit=32,swidth=32768,noquota 0 0
/dev/mapper/irphome_vg-imap_lv /mail xfs rw,relatime,attr2,delaylog,nobarrier,inode64,logbsize=256k,sunit=32,swidth=32768,usrquota,prjquota 0 0

cat /etc/mtab
/dev/mapper/irphome_vg-home_lv /home xfs rw,delaylog,inode64,nobarrier,logbsize=256k,uquota,prjquota 0 0
/dev/mapper/irphome_vg-imap_lv /mail xfs rw,delaylog,inode64,nobarrier,logbsize=256k,uquota,prjquota 0 0

List of details per wiki
#############
The most interesting thing in dmesg output was this:
XFS (dm-7): Failed to initialize disk quotas.  from /dev/disk/by-id dm-7 is my problem logical volume => dm-name-irphome_vg-home_lv -> ../../dm-7
#############

2.6.32-504.3.3.el6.x86_64

xfs_repair version 3.1.1

24 cpu using hyperthreading, so 12 real

mem
MemTotal:       49410148 kB
MemFree:          269628 kB
Buffers:          144256 kB
Cached:         47388884 kB
SwapCached:            0 kB
Active:           731016 kB
Inactive:       46871512 kB
Active(anon):       2976 kB
Inactive(anon):    71740 kB
Active(file):     728040 kB
Inactive(file): 46799772 kB
Unevictable:        5092 kB
Mlocked:            5092 kB
SwapTotal:      14331900 kB
SwapFree:       14331900 kB
Dirty:           3773708 kB
Writeback:             0 kB
AnonPages:         75696 kB
Mapped:           190092 kB
Shmem:               312 kB
Slab:            1012580 kB
SReclaimable:     875160 kB
SUnreclaim:       137420 kB
KernelStack:        5512 kB
PageTables:         9332 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    39036972 kB
Committed_AS:     293324 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      191424 kB
VmallocChunk:   34334431824 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6384 kB
DirectMap2M:     2080768 kB
DirectMap1G:    48234496 kB

/proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=24689396k,nr_inodes=6172349,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,barrier=1,data="" 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda1 /boot ext4 rw,relatime,barrier=1,data="" 0 0
/dev/mapper/irphome_vg-home_lv /home xfs rw,relatime,attr2,delaylog,nobarrier,inode64,logbsize=256k,sunit=32,swidth=32768,noquota 0 0
/dev/mapper/irphome_vg-imap_lv /mail xfs rw,relatime,attr2,delaylog,nobarrier,inode64,logbsize=256k,sunit=32,swidth=32768,usrquota,prjquota 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/mapper/homesavelv-homesavelv /homesave xfs rw,relatime,attr2,delaylog,sunit=32,swidth=32768,noquota 0 0

 /proc/partitions
major minor  #blocks  name

   8        0  143338560 sda
   8        1     512000 sda1
   8        2  142825472 sda2
   8       32 17179869184 sdc
   8       96 17179869184 sdg
   8      128 17179869184 sdi
   8       48 17179869184 sdd
   8      112 17179869184 sdh
   8       64 17179869184 sde
 253        0   52428800 dm-0
 253        1   14331904 dm-1
   8      160 17179869184 sdk
   8      176 17179869184 sdl
   8      192 17179869184 sdm
   8      224 17179869184 sdo
   8      240 17179869184 sdp
  65        0 17179869184 sdq
 253        3 17179869184 dm-3
 253        4 17179869184 dm-4
 253        5 17179869184 dm-5
 253        6 5368709120 dm-6
 253        7 42949672960 dm-7
   8       16 2147483648 sdb
   8       80 2147483648 sdf
 253        2 2147483648 dm-2
   8      144 2147483648 sdj
   8      208 2147483648 sdn
 253        8 2147467264 dm-8

 

 Raid layout
 3Par SAN  raid 6 12 4TB SAS disks (more or less, 3Par does some non-classic raid stuff)

 

 mpathd (360002ac000000000000000080000bf12) dm-4 3PARdata,VV
size=16T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 1:0:3:12 sdi 8:128 active ready running
  |- 2:0:0:12 sdm 8:192 active ready running
  |- 1:0:2:12 sde 8:64  active ready running
  `- 2:0:5:12 sdq 65:0  active ready running
mpathc (360002ac000000000000000070000bf12) dm-5 3PARdata,VV
size=16T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 1:0:2:11 sdd 8:48  active ready running
  |- 2:0:0:11 sdl 8:176 active ready running
  |- 1:0:3:11 sdh 8:112 active ready running
  `- 2:0:5:11 sdp 8:240 active ready running
mpathb (360002ac000000000000000060000bf12) dm-3 3PARdata,VV
size=16T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 1:0:2:10 sdc 8:32  active ready running
  |- 2:0:0:10 sdk 8:160 active ready running
  |- 1:0:3:10 sdg 8:96  active ready running
  `- 2:0:5:10 sdo 8:224 active ready running
mpathg (360002ac000000000000000110000bf12) dm-2 3PARdata,VV
size=2.0T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 1:0:2:1  sdb 8:16  active ready running
  |- 2:0:5:1  sdj 8:144 active ready running
  |- 1:0:3:1  sdf 8:80  active ready running
  `- 2:0:0:1  sdn 8:208 active ready running

pvscan
  PV /dev/mapper/mpathd   VG irphome_vg   lvm2 [16.00 TiB / 3.00 TiB free]
  PV /dev/mapper/mpathb   VG irphome_vg   lvm2 [16.00 TiB / 0    free]
  PV /dev/mapper/mpathc   VG irphome_vg   lvm2 [16.00 TiB / 0    free]
  PV /dev/mapper/mpathg   VG homesavelv   lvm2 [2.00 TiB / 0    free]
  PV /dev/sda2            VG VolGroup     lvm2 [136.21 GiB / 72.54 GiB free]
  Total: 5 [50.13 TiB] / in use: 5 [50.13 TiB] / in no VG: 0 [0   ]
 vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "irphome_vg" using metadata type lvm2
  Found volume group "homesavelv" using metadata type lvm2
  Found volume group "VolGroup" using metadata type lvm2
 lvscan
  ACTIVE            '/dev/irphome_vg/imap_lv' [5.00 TiB] inherit
  ACTIVE            '/dev/irphome_vg/home_lv' [40.00 TiB] inherit
  ACTIVE            '/dev/homesavelv/homesavelv' [2.00 TiB] inherit
  ACTIVE            '/dev/VolGroup/lv_root' [50.00 GiB] inherit
  ACTIVE            '/dev/VolGroup/lv_swap' [13.67 GiB] inherit

  

  lvdisplay irphome_vg/home_lv
  --- Logical volume ---
  LV Path                /dev/irphome_vg/home_lv
  LV Name                home_lv
  VG Name                irphome_vg
  LV UUID                8wLM12-e43p-UhIh-YTXn-kMBx-RffN-yNz2V5
  LV Write Access        read/write
  LV Creation host, time nuhome.irp.nia.nih.gov, 2014-12-01 17:53:47 -0500
  LV Status              available
  # open                 1
  LV Size                40.00 TiB
  Current LE             10485760
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:7

Disks, write cache etc are controlled by 3Par SAN, I just define up to 16TB blocks and export to host over FC or ISCSI.
In this case I am using FC.

 xfs_info /dev/irphome_vg/home_lv 
meta-data="" isize=256    agcount=40, agsize=268435452 blks
         =                       sectsz=512   attr=2, projid32bit=1
data     =                       bsize=4096   blocks=10737418080, imaxpct=5
         =                       sunit=4      swidth=4096 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=4 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_info /dev/irphome_vg/imap_lv 
meta-data="" isize=256    agcount=32, agsize=41943036 blks
         =                       sectsz=512   attr=2, projid32bit=1
data     =                       bsize=4096   blocks=1342177152, imaxpct=5
         =                       sunit=4      swidth=4096 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=4 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

dmesg output
SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
SGI XFS Quota Management subsystem
XFS (dm-7): delaylog is the default now, option is deprecated.
XFS (dm-7): Mounting Filesystem
XFS (dm-7): Ending clean mount
XFS (dm-7): Failed to initialize disk quotas.
XFS (dm-6): delaylog is the default now, option is deprecated.
XFS (dm-6): Mounting Filesystem
XFS (dm-6): Ending clean mount


scsi 2:0:0:0: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
scsi 2:0:0:10: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 2:0:0:0: [sdj] 4194304 512-byte logical blocks: (2.14 GB/2.00 GiB)
scsi 2:0:0:11: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 2:0:0:10: [sdk] 34359738368 512-byte logical blocks: (17.5 TB/16.0 TiB)
sd 2:0:0:0: [sdj] Write Protect is off
sd 2:0:0:0: [sdj] Mode Sense: 8b 00 10 08
scsi 2:0:0:12: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 2:0:0:11: [sdl] 34359738368 512-byte logical blocks: (17.5 TB/16.0 TiB)
sd 2:0:0:0: [sdj] Write cache: disabled, read cache: enabled, supports DPO and FUA
scsi 2:0:0:254: Enclosure         3PARdata SES              3210 PQ: 0 ANSI: 6
sd 2:0:0:12: [sdm] 34359738368 512-byte logical blocks: (17.5 TB/16.0 TiB)
scsi 2:0:1:0: RAID              HP       HSV400           0005 PQ: 0 ANSI: 5
scsi 2:0:2:0: RAID              HP       HSV400           0005 PQ: 0 ANSI: 5
 sdj:
sd 2:0:0:10: [sdk] Write Protect is off
sd 2:0:0:10: [sdk] Mode Sense: 8b 00 10 08
sd 2:0:0:11: [sdl] Write Protect is off
sd 2:0:0:11: [sdl] Mode Sense: 8b 00 10 08
scsi 2:0:3:0: RAID              HP       HSV400           0005 PQ: 0 ANSI: 5
 unknown partition table
sd 2:0:0:10: [sdk] Write cache: disabled, read cache: enabled, supports DPO and FUA
sd 2:0:0:11: [sdl] Write cache: disabled, read cache: enabled, supports DPO and FUA
scsi 2:0:4:0: RAID              HP       HSV400           0005 PQ: 0 ANSI: 5
sd 2:0:0:12: [sdm] Write Protect is off
sd 2:0:0:12: [sdm] Mode Sense: 8b 00 10 08
sd 2:0:0:12: [sdm] Write cache: disabled, read cache: enabled, supports DPO and FUA
scsi 2:0:5:0: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 2:0:5:0: [sdn] 4194304 512-byte logical blocks: (2.14 GB/2.00 GiB)
scsi 2:0:5:10: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 2:0:0:0: [sdj] Attached SCSI disk
scsi 2:0:5:11: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 2:0:5:0: [sdn] Write Protect is off
sd 2:0:5:0: [sdn] Mode Sense: 8b 00 10 08
SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
SGI XFS Quota Management subsystem
XFS (dm-7): delaylog is the default now, option is deprecated.
XFS (dm-7): Mounting Filesystem
XFS (dm-7): Ending clean mount
XFS (dm-7): Failed to initialize disk quotas.
XFS (dm-6): delaylog is the default now, option is deprecated.
XFS (dm-6): Mounting Filesystem
XFS (dm-6): Ending clean mount
Adding 14331900k swap on /dev/mapper/VolGroup-lv_swap.  Priority:-1 extents:1 across:14331900k 
device-mapper: table: 253:9: multipath: error getting device
device-mapper: ioctl: error adding target to table
pcc-cpufreq: (v1.00.00) driver loaded with frequency limits: 1600 MHz, 2400 MHz
sd 1:0:2:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatic
ally remap LUN assignments.
sd 2:0:0:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatic
ally remap LUN assignments.
sd 1:0:3:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatic
ally remap LUN assignments.
sd 2:0:5:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatic
ally remap LUN assignments.
device-mapper: multipath: Failing path 8:80.
device-mapper: multipath: Failing path 8:208.
device-mapper: multipath: Failing path 8:144.
end_request: I/O error, dev dm-2, sector 4194176
Buffer I/O error on device dm-2, logical block 524272
end_request: I/O error, dev dm-2, sector 4194176
Buffer I/O error on device dm-2, logical block 524272
end_request: I/O error, dev dm-2, sector 4194288
Buffer I/O error on device dm-2, logical block 524286
end_request: I/O error, dev dm-2, sector 4194288
Buffer I/O error on device dm-2, logical block 524286
end_request: I/O error, dev dm-2, sector 0
Buffer I/O error on device dm-2, logical block 0
end_request: I/O error, dev dm-2, sector 0
Buffer I/O error on device dm-2, logical block 0
end_request: I/O error, dev dm-2, sector 8
Buffer I/O error on device dm-2, logical block 1
end_request: I/O error, dev dm-2, sector 4194296
Buffer I/O error on device dm-2, logical block 524287
end_request: I/O error, dev dm-2, sector 4194296
Buffer I/O error on device dm-2, logical block 524287
end_request: I/O error, dev dm-2, sector 4194296
device-mapper: table: 253:2: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:2: multipath: error getting device
device-mapper: ioctl: error adding target to table
sd 1:0:3:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
sd 1:0:2:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
sd 2:0:5:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
sd 2:0:0:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
scsi 1:0:2:1: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 1:0:2:1: Attached scsi generic sg4 type 0
sd 1:0:2:1: [sdb] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
scsi 1:0:3:1: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 1:0:3:1: Attached scsi generic sg9 type 0
sd 1:0:3:1: [sdf] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
sd 1:0:2:1: [sdb] Write Protect is off
sd 1:0:2:1: [sdb] Mode Sense: 8b 00 10 08
sd 1:0:2:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
sd 1:0:3:1: [sdf] Write Protect is off
sd 1:0:3:1: [sdf] Mode Sense: 8b 00 10 08
sd 1:0:3:1: [sdf] Write cache: disabled, read cache: enabled, supports DPO and FUA
 sdb: unknown partition table
 sdf: unknown partition table
sd 1:0:2:1: [sdb] Attached SCSI disk
sd 1:0:3:1: [sdf] Attached SCSI disk
scsi 2:0:5:1: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 2:0:5:1: Attached scsi generic sg16 type 0
sd 2:0:5:1: [sdj] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
scsi 2:0:0:1: Direct-Access     3PARdata VV               3210 PQ: 0 ANSI: 6
sd 2:0:0:1: Attached scsi generic sg25 type 0
sd 2:0:0:1: [sdn] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
sd 2:0:5:1: [sdj] Write Protect is off
sd 2:0:5:1: [sdj] Mode Sense: 8b 00 10 08
sd 2:0:5:1: [sdj] Write cache: disabled, read cache: enabled, supports DPO and FUA
sd 2:0:0:1: [sdn] Write Protect is off
sd 2:0:0:1: [sdn] Mode Sense: 8b 00 10 08
sd 2:0:0:1: [sdn] Write cache: disabled, read cache: enabled, supports DPO and FUA
 sdj: unknown partition table
 sdn: unknown partition table
sd 2:0:5:1: [sdj] Attached SCSI disk
sd 2:0:0:1: [sdn] Attached SCSI disk
XFS (dm-8): Mounting Filesystem
XFS (dm-8): Ending clean mount
sd 1:0:2:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
sd 1:0:3:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
sd 2:0:5:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
sd 2:0:0:10: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
 rport-2:0-17: blocked FC remote port time out: removing rport
 rport-2:0-2: blocked FC remote port time out: removing rport

On Dec 22, 2014, at 3:48 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:

On Fri, Dec 19, 2014 at 09:26:12PM +0000, Weber, Charles (NIH/NIA/IRP) [E] wrote:
HI everyone, long time xfs/quota user with new server and problem
hardware is HP BL460 G7 blade, qlogic fiber channel and 3Par 7200 storage
3 16TB vols exported from 3Par to server via FC. These are thin volumes, but plenty of available backing storage.

Server runs current patched CentOS 6.6
kernel 2.6.32-504.3.3.el6.x86_64
xfsprogs 2.1.1-16.el6
Default mkfs.xfs options for volumes

mount options for logical volumes  home_lv 39TB imap_lv 4.6TB
/dev/mapper/irphome_vg-home_lv on /home type xfs (rw,delaylog,inode64,nobarrier,logbsize=256k,uquota,prjquota)
/dev/mapper/irphome_vg-imap_lv on /mail type xfs (rw,delaylog,inode64,nobarrier,logbsize=256k,uquota,prjquota)

Users are from large AD via winbind set to not enumerate. I saw
the bug with xfs_quota report not listing winbind defined user
names. Yes this happens to me.

So just enumerate them by uid. (report -un)

I can assign project quota to smaller volume. xfs_quota will not
report it. I cannot assign a project quota to larger volume. I get
this error: xfs_quota: cannot set limits: Function not
implemented.

You need to be more specific and document all your quota setup.

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

xfs_quota -x -c 'report -uh' /mail
User quota on /mail (/dev/mapper/irphome_vg-imap_lv)
                       Blocks
User ID      Used   Soft   Hard Warn/Grace
---------- ---------------------------------
root         2.2G      0      0  00 [------]

[xfs_quota -x -c 'report -uh' /home

nothing is returned

I can set user and project quotas on /mail but cannot see them. I have not tested them yet.
I cannot set user or project quotas on /home.
At one time I could definitely set usr quotas on /home. I did so and verified it worked.

Any ideas what is messed up on the /home volume?

Not without knowing a bunch more about your project quota setup.

Cheers,

Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx

<Prev in Thread] Current Thread [Next in Thread>