Hi guys, the earlier problem came out again:
Here are the logs:
[root]~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md0 42G 15G 26G 37% /
tmpfs 7.9G 0 7.9G 0% /lib/init/rw
udev 10M 76K 10M 1% /dev
tmpfs 7.9G 2.2M 7.9G 1% /dev/shm
/dev/md1 420G 31G 389G 8% /var/www/users
/dev/mapper/a-a 17T 17T 51G 100% /var/www/users/sms
[root]~# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 5.3M 152K 5.1M 3% /
tmpfs 2.0M 5 2.0M 1% /lib/init/rw
udev 2.0M 487 2.0M 1% /dev
tmpfs 2.0M 10 2.0M 1% /dev/shm
/dev/md1 420M 398K 419M 1% /var/www/users
/dev/mapper/a-a 203M 850K 202M 1% /var/www/users/sms
[root]~# xfs_info /var/www/users/sms
meta-data=/dev/a/a isize=256 agcount=32, agsize=137326144
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=4394436608, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
[root]~# xfs_logprint /dev/a/a
xfs_logprint:
xfs_logprint: /dev/a/a contains a mounted and writable filesystem
data device: 0xfd00
log device: 0xfd00 daddr: 17577746464 length: 262144
Header 0x1 wanted 0xfeedbabe
**********************************************************************
* ERROR: header cycle=1 block=176303 *
**********************************************************************
Bad log record header
[root]~# umount /var/www/users/sms
[root]~# xfs_repair /dev/a/a
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- clear lost+found (if it exists) ...
- clearing existing "lost+found" inode
- deleting existing "lost+found" entry
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- ensuring existence of lost+found directory
- traversing filesystem starting at / ...
- traversal finished ...
- traversing all unattached subtrees ...
- traversals finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
[root]~# mount /var/www/users/sms
[root]~# touch /var/www/users/sms/1
touch: cannot touch `/var/www/users/sms/1': No space left on device
[root]~# xfs_logprint /dev/a/a
xfs_logprint:
xfs_logprint: /dev/a/a contains a mounted and writable filesystem
data device: 0xfd00
log device: 0xfd00 daddr: 17577746464 length: 262144
cycle: 1 version: 1 lsn: 1,0 tail_lsn: 1,0
length of Log Record: 20 prev offset: -1 num ops: 1
uuid: 68b5aa39-c15e-4170-a43f-9278e95911ea format: little endian linux
h_size: 32768
----------------------------------------------------------------------------
Oper (0): tid: b0c0d0d0 len: 8 clientid: LOG flags: UNMOUNT
Unmount filesystem
============================================================================
xfs_logprint: skipped 512 cleared blocks in range: 2 - 513
xfs_logprint: skipped 261630 zeroed blocks in range: 514 - 262143
xfs_logprint: physical end of log
============================================================================
xfs_logprint: logical end of log
============================================================================
We have another server, which has exactly the same hardware architecture,
exactly the same system, it's the same server we use dns load balancing
between the two, and there we have no problem at all, but see the log:
[root]~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md0 42G 12G 30G 28% /
tmpfs 7.9G 0 7.9G 0% /lib/init/rw
udev 10M 72K 10M 1% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/mapper/a-a 17T 17T 28G 100% /var/www/users/sms
[root]~# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 5.3M 293K 5.0M 6% /
tmpfs 2.0M 5 2.0M 1% /lib/init/rw
udev 2.0M 469 2.0M 1% /dev
tmpfs 2.0M 1 2.0M 1% /dev/shm
/dev/mapper/a-a 309M 857K 309M 1% /var/www/users/sms
[root]~# xfs_info /var/www/users/sms
meta-data=/dev/a/a isize=256 agcount=32, agsize=137326144
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=4394436608, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
[root]~# xfs_logprint /dev/a/a
xfs_logprint:
xfs_logprint: /dev/a/a contains a mounted and writable filesystem
data device: 0xfd00
log device: 0xfd00 daddr: 17577746464 length: 262144
Header 0x3 wanted 0xfeedbabe
**********************************************************************
* ERROR: header cycle=3 block=55900 *
**********************************************************************
Bad log record header
So you tell me, it appeared again just suddenly.
Thank you,
Viktor
-----Original Message-----
From: David Chinner [mailto:dgc@xxxxxxx]
Sent: Tuesday, April 01, 2008 2:35 AM
To: Huszár Viktor Dénes
Cc: 'Emmanuel Florac'; xfs@xxxxxxxxxxx
Subject: Re: free space problem
On Mon, Mar 31, 2008 at 07:36:56PM +0200, Huszár Viktor Dénes wrote:
> -----Original Message-----
> From: Emmanuel Florac [mailto:eflorac@xxxxxxxxxxxxxx]
> Sent: Monday, March 31, 2008 9:40 AM
> To: Huszár Viktor Dénes; xfs@xxxxxxxxxxx
> Subject: Re: free space problem
>
> Le Mon, 31 Mar 2008 02:31:16 +0200 vous écriviez:
>
> >Then maybe you simply ran out of inodes. It's common if you have lots
> >of small files. There is a way to increase the number of inodes but I
> >don't remember it right now.
> --
> No, unfortunately not. There are not many small files and the inode usage
is
> 1%. See.
Yes, but if you have fragemented free space then it is possible that there
are not enough free extents large enough (or aligned correctly) to
allocate more inodes. The number of "free inodes" reported doesn't take
this into account; it only looks at the number of free blocks and converts
that to a theoretical number of inodes that could be allocated in that
space (i.e. it assumes perfect fit and no waste).
In this "not quite full filesystem" situation, you can write data to the
filesystem, but any attempt to create a new inode (new file, directory,
etc) will fail with ENOSPC. This sounds like the symptoms you are
reporting....
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
__________ Information from ESET NOD32 Antivirus, version of virus signature
database 2988 (20080331) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
|